Stephen Reesehttps://www.rsreese.com/2018-01-15T12:00:00-05:00Network Traffic Capture in Virtual Enviroments2018-01-15T12:00:00-05:002018-01-15T12:00:00-05:00Stephen Reesetag:www.rsreese.com,2018-01-15:/network-traffic-capture-in-virtual-enviroments/This post demonstrates how you mirror interfaces on a virtual private server (VPS) in a cloud environment, e.g. virtual machine (VM) on a hypervisor where you do not have access to network or virtualization infrastructure where a network TAP or SPAN port would be available. This technique is used …<p>This post demonstrates how you mirror interfaces on a virtual private server (<span class="caps">VPS</span>) in a cloud environment, e.g. virtual machine (<span class="caps">VM</span>) on a hypervisor where you do not have access to network or virtualization infrastructure where a network <span class="caps">TAP</span> or <span class="caps">SPAN</span> port would be available. This technique is used to forward packets to a collection point for aggregation and/or analysis. A scenario may be monitoring network traffic for security threats with a central security stack running tools such a Snort, Suricata and/or Bro <span class="caps">IDS</span>. Example cloud providers are Linode, Digital Ocean and <span class="caps">AWS</span>.</p>
<p>While a single network interface will work and is used in our examples, the client node being monitored should have two network interfaces, one for production traffic and the second interface for sending traffic to your collection node, e.g. your cloud based security stack or where you want to store the packet captures. This is for performance reasons as you are essentially doubling the traffic on a single interface. You will need to be cognizant of the amount of data you are sending to your aggregation point (collection node) as it may become saturated as well if you send traffic from too many client nodes that exceed the collection node interface capacity. Sending traffic from 20 client nodes with 1Gbs interfaces to one capture node that has a 10Gbs will obviously drop packets depending on how much traffic is being forwarding from clients. Note that many providers do provide greater bandwidth internally, e.g. support 1Gbs public interfaces but 10+Gbs internally. Another mitigation would be shape the traffic using <code>tc</code> or something similar in order minimize this from the client nodes. You must also consider either encrypting the tunnel using IPSec or using a trusted transport network. We do not address the security or performance implications in this post but instead its implementation.</p>
<p>We will provide three examples using IPTables and two using tc (Traffic Control) over both <span class="caps">VXLAN</span> and <span class="caps">GRE</span> tunnels. The examples are performed on Ubuntu 16.04 hosts in <span class="caps">AWS</span>. From my experiments, I found <span class="caps">VXLAN</span> (example four) to be quite useful in that I did not have to specify remote endpoints on the collection node. This allows multiple clients to forward traffic over a multiple tunnels to one collection node interface which allows for easy capture and analysis. <span class="caps">GRE</span> tunnels are point-to-point which make capture and aggregation a difficult task for many client nodes which result in an interface per tunnel. If you are aware of a workaround for this, please let me know.</p>
<p>The first example is the easiest to configure but has a caveat that <span class="caps">MAC</span> addresses will appear from the client tunnel interface verse the actual source interface due to IPTables. This may be okay for one off usage but if using for a large deployment you will likely want the hardware address for performing analysis and traceability from the interface traffic is traversing verse having to track which virtual interface is associated with which client node.</p>
<p>Create <span class="caps">VXLAN</span> tunnel on collection node. <span class="caps">VXLAN</span> is used in this example but we will provide a second IPTables example where <span class="caps">GRE</span> is used</p>
<div class="highlight"><pre><span></span>ip link add name vxlan42 type vxlan id 42 dev eth0 local 172.31.108.76 dstport 4789
ip address add 172.20.100.10/24 dev vxlan42
ip link set up vxlan42
</pre></div>
<p>Create <span class="caps">VXLAN</span> tunnel on client to collection node</p>
<div class="highlight"><pre><span></span>ip link add name vxlan42 type vxlan id 42 dev eth0 local 172.31.102.153 remote 172.31.108.76 dstport 4789
ip address add 172.20.100.1/24 dev vxlan42
ip link set up vxlan42
</pre></div>
<p>Use IPTables on client node to forward traffic over tunnel to the collection node</p>
<div class="highlight"><pre><span></span>iptables -I PREROUTING -t mangle -j TEE --gateway 172.20.100.10
iptables -I POSTROUTING -t mangle -j TEE --gateway 172.20.100.10
iptables -A POSTROUTING -t mangle -p tcp --tcp-flags SYN,RST SYN -o tun0 -j TCPMSS --clamp-mss-to-pmtu
</pre></div>
<p>On the collection node you will now see all the traffic traversing eth0 on the client node using a tool such as tcpdump, e.g. <code>tcpdump -i tun0 -en</code>. You can filter using IPTables on the client node in order to reduce traffic sent to collection node, e.g. only send traffic you care about storing or analyzing.</p>
<p>The second example uses <code>gretap</code> <span class="caps">GRE</span> tunnel but we have to establish a point-to-point link which requires multiple interfaces on the collection node if we want to support multiple client nodes. As you can imagine, if you had ten client nodes you were trying to capture from, you need to listen to ten interfaces, not a great solution for security monitoring. This solution allows us to maintain the <span class="caps">MAC</span> header over a <span class="caps">GRE</span> tunnel but in this example, we are still using IPTables to forward traffic over the tunnel therefore the <span class="caps">MAC</span> header is still associated with the tunnel verse actual interface as discussed in the first example.</p>
<p>Create <span class="caps">GRE</span> tunnel on collection node</p>
<div class="highlight"><pre><span></span>ip link add tun0 type gretap local 172.31.108.76 remote 172.31.102.153
ip link set tun0 up
ip addr add 172.20.100.10/24 dev tun0
</pre></div>
<p>Create tunnel on client to collection node</p>
<div class="highlight"><pre><span></span>ip link add tun0 type gretap local 172.31.102.153 remote 172.31.108.76
ip link set tun0 up
ip addr add 172.20.100.2/24 dev tun0
</pre></div>
<p>Use IPTables on client node to forward traffic over tunnel to the collection node</p>
<div class="highlight"><pre><span></span>iptables -I PREROUTING -t mangle -j TEE --gateway 172.20.100.10
iptables -I POSTROUTING -t mangle -j TEE --gateway 172.20.100.10
iptables -A POSTROUTING -t mangle -p tcp --tcp-flags SYN,RST SYN -o tun0 -j TCPMSS --clamp-mss-to-pmtu
</pre></div>
<p>The third example uses an <code>ip tunnel</code> <span class="caps">GRE</span> point-to-point link which requires multiple interfaces on the collection node if we want to support multiple client nodes just as the case in the above <code>gretap</code> example. I am including this as some folks may not care about including the <span class="caps">MAC</span> header and the lack of it may provide a small performance improvement as the overall packet size is reduced.</p>
<p>Create <span class="caps">GRE</span> tunnel on collection node</p>
<div class="highlight"><pre><span></span>modprobe ip_gre
lsmod | grep ip_gre
ip tunnel add tun0 mode gre local 172.31.108.76 remote 172.31.102.153 ttl 255
ip link set tun0 up
ip addr add 172.20.100.10/24 dev tun0
</pre></div>
<p>Create tunnel on client to collection node</p>
<div class="highlight"><pre><span></span>modprobe ip_gre
lsmod | grep ip_gre
ip tunnel add tun0 mode gre local 172.31.102.153 remote 172.31.108.76 ttl 255
ip link set tun0 up
ip addr add 172.20.100.2/24 dev tun0
</pre></div>
<p>Use IPTables on client node to forward traffic over tunnel to the collection node</p>
<div class="highlight"><pre><span></span>iptables -I PREROUTING -t mangle -j TEE --gateway 172.20.100.10
iptables -I POSTROUTING -t mangle -j TEE --gateway 172.20.100.10
iptables -A POSTROUTING -t mangle -p tcp --tcp-flags SYN,RST SYN -o tun0 -j TCPMSS --clamp-mss-to-pmtu
</pre></div>
<p>The fourth example uses <code>tc</code> in order to capture and forward traffic. <code>tc</code> offers a very rich set of tools for managing and manipulating the transmission of packets. We can forward packets or flows of our choice over the tunnel to the analysis node. In researching how to setup remote sensors in cloud computing environments, I learned that <code>tc</code> will not readily forward egress traffic over a tunnel interface. The solution is to forward the traffic we care about to our loopback adapter, then forward the ingress loopback traffic flow to the tunnel so we are then able to see the ingress and egress packets on our collection node. The use of <code>tc</code> allows us to maintain our original <span class="caps">MAC</span> header where as IPTables did not. For this example we start again using <span class="caps">VXLAN</span> which allows us to send multiple client tunnels to one interface on our collection node. A win for easily aggregating and analyzing traffic from multiple client nodes on one collection node.</p>
<p>Capture node</p>
<div class="highlight"><pre><span></span>ip link add name vxlan42 type vxlan id 42 dev eth0 local 172.31.108.76 dstport 4789
ip address add 172.20.100.10/24 dev vxlan42
ip link set up vxlan42
</pre></div>
<p>Sending node</p>
<div class="highlight"><pre><span></span>ip link add name vxlan42 type vxlan id 42 dev eth0 local 172.31.102.153 remote 172.31.108.76 dstport 4789
ip address add 172.20.100.2/24 dev vxlan42
ip link set up vxlan42
</pre></div>
<p>Send ingress traffic to tunnel</p>
<div class="highlight"><pre><span></span>tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: \
protocol all \
u32 match u8 0 0 \
action mirred egress mirror dev vxlan42
</pre></div>
<p>Since loops are not hard to create in the egress qdiscs, we push to loopback and then the tunnel</p>
<div class="highlight"><pre><span></span>tc qdisc add dev eth0 handle 1: root prio
tc filter add dev eth0 parent 1: \
protocol all \
u32 match u8 0 0 \
action mirred egress mirror dev lo
</pre></div>
<p>Select all traffic </p>
<div class="highlight"><pre><span></span>tc qdisc add dev lo ingress
tc filter add dev lo parent ffff: \
protocol all u32 \
match u8 0 0 \
action mirred egress mirror dev vxlan42
</pre></div>
<p>Then drop <span class="caps">VXLAN</span> traffic so we do not see it again on the collection node</p>
<div class="highlight"><pre><span></span>tc filter add dev lo parent ffff: \
protocol ip u32 \
match ip dst 172.31.108.76/32 \
match ip dport 4789 0xffff \
action drop
</pre></div>
<p>The fifth and last example uses <code>gretap</code> along with <code>tc</code>. This allows us to maintain the <span class="caps">MAC</span> header over a <span class="caps">GRE</span> tunnel but in this example, remember we are still using IPTables therefore the <span class="caps">MAC</span> header is still associated with the tunnel verse actual interface.</p>
<p>Create <span class="caps">GRE</span> tunnel on collection node</p>
<div class="highlight"><pre><span></span>ip link add tun0 type gretap local 172.31.108.76 remote 172.31.102.153
ip link set tun0 up
ip addr add 172.20.100.10/24 dev tun0
</pre></div>
<p>Create tunnel on client to collection node</p>
<div class="highlight"><pre><span></span>ip link add tun0 type gretap local 172.31.102.153 remote 172.31.108.76
ip link set tun0 up
ip addr add 172.20.100.2/24 dev tun0
</pre></div>
<p>Send ingress traffic to tunnel</p>
<div class="highlight"><pre><span></span>tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: \
protocol all \
u32 match u8 0 0 \
action mirred egress mirror dev tun0
</pre></div>
<p>Since loops are not hard to create in the egress qdiscs, we push to loopback and then the tunnel</p>
<div class="highlight"><pre><span></span>tc qdisc add dev eth0 handle 1: root prio
tc filter add dev eth0 parent 1: \
protocol all \
u32 match u8 0 0 \
action mirred egress mirror dev lo
</pre></div>
<p>Select all traffic </p>
<div class="highlight"><pre><span></span>tc qdisc add dev lo ingress
tc filter add dev lo parent ffff: \
protocol all u32 \
match u8 0 0 \
action mirred egress mirror dev tun0
</pre></div>
<p>Then drop <span class="caps">GRE</span> traffic so we do not see it again on the collection node</p>
<div class="highlight"><pre><span></span> tc filter add dev lo parent ffff: \
protocol ip u32 \
match ip dst 172.31.108.76/32 \
match ip protocol 0x2f 0xff \
action drop
</pre></div>
<p>There you have it. Please leave a comment if you have any questions.</p>Network Traffic Capture on Linux using OpenvSwitch2017-10-25T12:00:00-04:002017-10-25T12:00:00-04:00Stephen Reesetag:www.rsreese.com,2017-10-25:/network-traffic-capture-on-linux-using-openvswitch/This post demonstrates how you can mirror interfaces on a Linux server in an environment where you may not have physical network taps or SPAN ports. We can use OpenvSwitch in order to forward traffic between nodes, even if we are not using virtualization. Each node being monitored needs two …<p>This post demonstrates how you can mirror interfaces on a Linux server in an environment where you may not have physical network taps or <span class="caps">SPAN</span> ports. We can use <a href="http://docs.openvswitch.org/en/latest/howto/tunneling/">OpenvSwitch</a> in order to forward traffic between nodes, even if we are not using virtualization. Each node being monitored needs two interfaces, one for production traffic and the other being an internal or mirrored interface where you send traffic to be aggregated and analyzed by your cloud based security stack. You will need to be cognizant of the amount of data you are sending to your aggregation point as it may become saturated if you send traffic from multiple nodes that exceeds the receiving nodes capacity.</p>
<p>On <span class="caps">VM</span> to have a monitored interface:</p>
<p>Ensure the host has two network interfaces and determine which one is production verse management. The management interface will be used to send traffic to your aggregation or collection node as previously described above. For this example, eth0 and eth1 are production and management respectively.</p>
<p>Install OpenvSwitch:</p>
<div class="highlight"><pre><span></span>$ sudo apt-get install openvswitch-switch
</pre></div>
<p>Bring up the secondary interface, we will use this as the bridge interface, i.e. the interface that sends mirrored eth0 traffic:</p>
<div class="highlight"><pre><span></span>$ sudo ifconfig eth1 <span class="m">172</span>.31.3.110 netmask <span class="m">255</span>.255.240.0
</pre></div>
<p>Configure bridge and set remote <span class="caps">IP</span> to your collection node which is a different network (interface) then that which is being mirrored:</p>
<div class="highlight"><pre><span></span>$ sudo ovs-vsctl add-br br0
$ sudo ovs-vsctl add-port br0 eth1
$ sudo ovs-vsctl add-port br0 gre0 -- <span class="nb">set</span> interface gre0 <span class="nv">type</span><span class="o">=</span>gre options:remote_ip<span class="o">=</span><span class="m">172</span>.31.10.151 -- --id<span class="o">=</span>@p get port gre0 -- --id<span class="o">=</span>@m create mirror <span class="nv">name</span><span class="o">=</span>m0 <span class="k">select</span>-all<span class="o">=</span><span class="nb">true</span> output-port<span class="o">=</span>@p -- <span class="nb">set</span> bridge br0 <span class="nv">mirrors</span><span class="o">=</span>@m
</pre></div>
<p>The following steps will disconnect you from eth0 so may be ideal to connect to eth1 at this point or respectively your bridge interface. Null the network address to be mirrored and set the <span class="caps">IP</span> to that of the bridge interface as well as updating the gateway. We also assign the bridge interface to the <span class="caps">MAC</span> address of eth0 as some environments may not allow traffic to/from interfaces hardware addresses they do not know about.</p>
<div class="highlight"><pre><span></span>$ sudo ifconfig br0 <span class="m">172</span>.31.11.64 netmask <span class="m">255</span>.255.240.0
$ sudo ifconfig eth0 <span class="m">0</span>
$ sudo ifconfig br0 hw ether 0a:74:0c:89:fb:70
$ sudo route add default gw <span class="m">172</span>.31.0.1 br0
</pre></div>
<p>We can now view the mirrored traffic on the host defined at the remote <span class="caps">IP</span>, packets are encapsulated but you may see protocol unreachable <span class="caps">ICMP</span> messages. This is because br0 drops responses. The next step fixes this by completing/terminating the tunnel on the remote host which will unencapsulate the <span class="caps">GRE</span> tunnel. Here, we again use eth0 and eth1 as production and management networks but we do not have to. We could just have one interface that accepts traffic from the clients forwarding us their network traffic but if it becomes saturated it may be difficult to connect to the host.</p>
<div class="highlight"><pre><span></span>$ sudo ifconfig eth1 <span class="m">172</span>.20.1.7 netmask <span class="m">255</span>.255.255.240
$ sudo modprobe ip_gre
$ sudo lsmod <span class="p">|</span> grep ip_gre
$ sudo ip tunnel add mon0 mode gre <span class="nb">local</span> <span class="m">172</span>.20.1.7 remote
$ sudo ip addr add <span class="m">1</span>.1.1.1/30 dev mon0
$ sudo ip link <span class="nb">set</span> mon0 up
</pre></div>
<p>Now you can monitor interface mon0 using tools like tcpdump or simply capture network traffic for retroactive analysis.</p>
<p>If you need to, remove the bridge and port using the following commands:</p>
<div class="highlight"><pre><span></span>$ sudo ovs-vsctl clear bridge br0 mirrors
$ sudo ovs-vsctl del-port br0 gre0
</pre></div>Benchmarking Websites with ab and tsung2017-10-10T12:00:00-04:002017-10-10T12:00:00-04:00Stephen Reesetag:www.rsreese.com,2017-10-10:/benchmarking-websites-with-ab-and-tsung/Everyone enjoys responsive websites and being that I host a few, look for ways to improve their speed. Previously, I was interested in, HTTP, HTTPS, and HTTP/WAF, I now primarily focus on HTTPS. Browsers and third-party online services may be used in order to benchmark page performance but began …<p>Everyone enjoys responsive websites and being that I host a few, look for ways to improve their speed. Previously, I was interested in, <span class="caps">HTTP</span>, <span class="caps">HTTPS</span>, and <span class="caps">HTTP</span>/<span class="caps">WAF</span>, I now primarily focus on <span class="caps">HTTPS</span>. Browsers and third-party online services may be used in order to benchmark page performance but began to look at other solutions. Two online services are <a href="http://tools.pingdom.com/fpt/">Pingdom Website Speed Test</a> and <a href="https://developers.google.com/speed/pagespeed/insights/">PageSpeed Insights</a>.</p>
<p>The first tool I leveraged was Apache Bench, commonly known as <code>ab</code>. This allows me to run a quick test in order to determine the max requests per second (req/s). While fun, it is not a practical metric as there a a number of factors that must be considered when benchmarking a web-service and understanding where weaknesses may present themselves.</p>
<p><span class="caps">HTTPS</span> requests with keep-alives, connection reuse provides significant speedup:</p>
<div class="highlight"><pre><span></span>$ ab -k -n <span class="m">60000</span> -c <span class="m">100</span> -f TLS1.2 -H <span class="s2">"Accept-Encoding: gzip,deflate"</span> https://www.rsreese.com/web-stack/
This is ApacheBench, Version <span class="m">2</span>.3 <<span class="nv">$Revision</span>: <span class="m">1757674</span> $>
Copyright <span class="m">1996</span> Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Server Software: nginx
Server Hostname: www.rsreese.com
Server Port: <span class="m">443</span>
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
TLS Server Name: www.rsreese.com
Document Path: /web-stack/
Document Length: <span class="m">2575</span> bytes
Concurrency Level: <span class="m">100</span>
Time taken <span class="k">for</span> tests: <span class="m">7</span>.124 seconds
Complete requests: <span class="m">60000</span>
Failed requests: <span class="m">0</span>
Keep-Alive requests: <span class="m">59447</span>
Total transferred: <span class="m">220557235</span> bytes
HTML transferred: <span class="m">154500000</span> bytes
Requests per second: <span class="m">8422</span>.57 <span class="o">[</span><span class="c1">#/sec] (mean)</span>
Time per request: <span class="m">11</span>.873 <span class="o">[</span>ms<span class="o">]</span> <span class="o">(</span>mean<span class="o">)</span>
Time per request: <span class="m">0</span>.119 <span class="o">[</span>ms<span class="o">]</span> <span class="o">(</span>mean, across all concurrent requests<span class="o">)</span>
Transfer rate: <span class="m">30235</span>.32 <span class="o">[</span>Kbytes/sec<span class="o">]</span> received
Connection Times <span class="o">(</span>ms<span class="o">)</span>
min mean<span class="o">[</span>+/-sd<span class="o">]</span> median max
Connect: <span class="m">0</span> <span class="m">1</span> <span class="m">8</span>.0 <span class="m">0</span> <span class="m">172</span>
Processing: <span class="m">0</span> <span class="m">11</span> <span class="m">5</span>.5 <span class="m">11</span> <span class="m">55</span>
Waiting: <span class="m">0</span> <span class="m">11</span> <span class="m">5</span>.4 <span class="m">10</span> <span class="m">45</span>
Total: <span class="m">0</span> <span class="m">12</span> <span class="m">10</span>.5 <span class="m">11</span> <span class="m">203</span>
Percentage of the requests served within a certain <span class="nb">time</span> <span class="o">(</span>ms<span class="o">)</span>
<span class="m">50</span>% <span class="m">11</span>
<span class="m">66</span>% <span class="m">13</span>
<span class="m">75</span>% <span class="m">15</span>
<span class="m">80</span>% <span class="m">16</span>
<span class="m">90</span>% <span class="m">18</span>
<span class="m">95</span>% <span class="m">21</span>
<span class="m">98</span>% <span class="m">26</span>
<span class="m">99</span>% <span class="m">28</span>
<span class="m">100</span>% <span class="m">203</span> <span class="o">(</span>longest request<span class="o">)</span>
</pre></div>
<p>While Apache Bench provides a quick analysis of some of our page speed, <code>tsung</code> is benchmark tool that can provide additional performance insights through its advanced configuration options. The configuration states that we are running <code>tsung</code> locally, the target host, the interval for this phase (yes, you can have more), user agent in which we have two with a ratio defined, and finally the session, which in this case will cause <code>tsung</code> to send as many requests as it can. Again, this is not realistic, just fun.</p>
<div class="highlight"><pre><span></span><span class="cp"><?xml version="1.0"?></span><span class="nt"><tsung</span> <span class="na">loglevel=</span><span class="s">"notice"</span> <span class="na">version=</span><span class="s">"1.0"</span><span class="nt">></span>
<span class="nt"><clients></span>
<span class="nt"><client</span> <span class="na">host=</span><span class="s">"localhost"</span> <span class="na">use_controller_vm=</span><span class="s">"true"</span> <span class="na">maxusers=</span><span class="s">"10000"</span><span class="nt">/></span>
<span class="nt"></clients></span>
<span class="nt"><servers></span>
<span class="nt"><server</span> <span class="na">host=</span><span class="s">"www.rsreese.com"</span> <span class="na">port=</span><span class="s">"443"</span> <span class="na">type=</span><span class="s">"ssl"</span><span class="nt">/></span>
<span class="nt"></servers></span>
<span class="nt"><load></span>
<span class="nt"><arrivalphase</span> <span class="na">phase=</span><span class="s">"1"</span> <span class="na">duration=</span><span class="s">"1"</span> <span class="na">unit=</span><span class="s">"minute"</span><span class="nt">></span>
<span class="nt"><users</span> <span class="na">maxnumber=</span><span class="s">"10000"</span> <span class="na">interarrival=</span><span class="s">"0.05"</span> <span class="na">unit=</span><span class="s">"second"</span><span class="nt">/></span>
<span class="nt"></arrivalphase></span>
<span class="nt"></load></span>
<span class="nt"><options></span>
<span class="nt"><option</span> <span class="na">type=</span><span class="s">"ts_http"</span> <span class="na">name=</span><span class="s">"user_agent"</span><span class="nt">></span>
<span class="nt"><user_agent</span> <span class="na">probability=</span><span class="s">"80"</span><span class="nt">></span>Mozilla/5.0 (Windows NT 6.1; rv:34.0) Gecko/20100101 Firefox/34.0<span class="nt"></user_agent></span>
<span class="nt"><user_agent</span> <span class="na">probability=</span><span class="s">"20"</span><span class="nt">></span>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36<span class="nt"></user_agent></span>
<span class="nt"></option></span>
<span class="nt"></options></span>
<span class="nt"><sessions></span>
<span class="nt"><session</span> <span class="na">name=</span><span class="s">"web-stack"</span> <span class="na">probability=</span><span class="s">"100"</span> <span class="na">type=</span><span class="s">"ts_http"</span><span class="nt">></span>
<span class="nt"><for</span> <span class="na">from=</span><span class="s">"1"</span> <span class="na">to=</span><span class="s">"10000"</span> <span class="na">var=</span><span class="s">"i"</span><span class="nt">></span>
<span class="nt"><request><http</span> <span class="na">url=</span><span class="s">"/web-stack/"</span> <span class="na">version=</span><span class="s">"1.1"</span> <span class="na">method=</span><span class="s">"GET"</span><span class="nt">/></request></span>
<span class="nt"></for></span>
<span class="nt"></session></span>
<span class="nt"></sessions></span>
<span class="nt"></tsung></span>
</pre></div>
<p>Run <code>tsung</code> and generate the reports. Optionaly, multiple reports can be combined. You may have to sudo depending your systems permissions.</p>
<div class="highlight"><pre><span></span>$ tsung -f origin.xml start
$ <span class="nb">cd</span> results-directory
$ /usr/lib/tsung/bin/tsung_stats.pl
$ tsplot <span class="s2">"HTTP"</span> <span class="m">20150418</span>-1658/tsung.log <span class="s2">"HTTPS"</span> <span class="m">20150418</span>-1712/tsung.log -d combine2/
</pre></div>
<p><code>tsung</code> provides useful reports and graphics. For the sake of brivety, I will not include the report but just a few charts.</p>
<p><a href="https://www.rsreese.com/assets/request_count.png"><img alt="Request Count" src="https://www.rsreese.com/assets/request_count.png"></a></p>
<p><a href="https://www.rsreese.com/assets/request_mean.png"><img alt="Request Mean" src="https://www.rsreese.com/assets/request_mean.png"></a></p>
<p><a href="https://www.rsreese.com/assets/size_rcv.png"><img alt="Received Size" src="https://www.rsreese.com/assets/size_rcv.png"></a></p>
<p><a href="https://www.rsreese.com/assets/size_sent.png"><img alt="Sent Size" src="https://www.rsreese.com/assets/size_sent.png"></a></p>
<p>With this baseline, you can tailor the <code>tsung</code> configuration to include phases of increasing user load along with multiple pages and actions. See the <code>tsung</code> <a href="http://tsung.erlang-projects.org/user_manual/">documention</a> for details and leave a comment below if you have any questions about this post.</p>Detecting Tor traffic with Bro network traffic analyzer2016-01-16T12:00:00-05:002016-01-16T12:00:00-05:00Stephen Reesetag:www.rsreese.com,2016-01-16:/detecting-tor-traffic-with-bro-network-traffic-analyzer/This entry is a post in a series in order to identify Tor (the onion router) network traffic and usage using Bro Network Security Monitor. To learn more about both projects, please visit the aforementioned links. This post is not to argue the merits of allowing Tor to run on …<p>This entry is a post in a <a href="http://www.rsreese.com/tag/tor/">series</a> in order to identify <a href="https://www.torproject.org/about/overview.html.en">Tor</a> (the onion router) network traffic and usage using <a href="https://www.bro.org/sphinx/intro/index.html">Bro Network Security Monitor</a>. To learn more about both projects, please visit the aforementioned links. This post is not to argue the merits of allowing Tor to run on a network. Due to malware variants taking advantage of Tor for its <a href="http://threatpost.com/huge-botnet-found-using-tor-network-for-communications/102179">botnet</a> command and control (C2), I wanted to be able to effectively identify Tor usage in hopes of identifying hosts that may be using Tor for C2 purposes.</p>
<p>A method folks often use to identify communication with Tor relays is to compare the current list of known Tor <a href="https://www.dan.me.uk/torlist/">servers</a> with the traffic from their network. While this does work, some relays may host other legitimate services which could introduce false-positives. The goal was to find a method to augment the parsing network traffic for Tor server matches which is sometimes done retrospectively.</p>
<p>If we take a look at the Tor certificates, we see an interesting pattern for the Issuer and Subject <span class="caps">ID</span> form a pattern.</p>
<p><a href="https://www.rsreese.com/assets/tor-wireshark.png"><img alt="Screen Shot" src="https://www.rsreese.com/assets/tor-wireshark-thumb.png"></a></p>
<p>Using tshark, it the Issuer and Subject <span class="caps">ID</span> patterns are a little more apparent.</p>
<div class="highlight"><pre><span></span>$ tshark -r tor.pcap -T fields -R <span class="s2">"ssl.handshake.certificate"</span> -e x509af.utcTime -e x509sat.uTF8String
<span class="m">13</span>-10-15 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-02-11 <span class="m">23</span>:59:59 <span class="o">(</span>UTC<span class="o">)</span> www.axslhtfqq.com,www.hkkch64skp7am.net
<span class="m">13</span>-12-30 <span class="m">18</span>:32:48 <span class="o">(</span>UTC<span class="o">)</span>,14-12-30 <span class="m">18</span>:32:48 <span class="o">(</span>UTC<span class="o">)</span> www.igdpzct5tauwgyqs.com,www.4tdznzbrfuv.net
<span class="m">13</span>-10-04 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-04-22 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.3pxivyds.com,www.nolspqtib3ix.net
<span class="m">13</span>-11-17 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-06-22 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.3pzqe4en5.com,www.glk3fwiz6.net
<span class="m">13</span>-06-19 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-04-20 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.5orbut4ufhohm5rlj47.com,www.orutxjqwf.net
<span class="m">13</span>-06-15 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-02-04 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.7wdf4rkj5mew.com,www.sd5mkmsmo.net
<span class="m">13</span>-11-19 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-02-05 <span class="m">23</span>:59:59 <span class="o">(</span>UTC<span class="o">)</span> www.75ba5lymxpbhw3a2kb.com,www.rnspic4yus5crf6w.net
<span class="m">13</span>-12-30 <span class="m">19</span>:54:02 <span class="o">(</span>UTC<span class="o">)</span>,14-12-30 <span class="m">19</span>:54:02 <span class="o">(</span>UTC<span class="o">)</span> www.s5rc22gpzrwt4e.com,www.qzsg2ioaoplbs2gaha5.net
<span class="m">13</span>-08-12 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-04-16 <span class="m">23</span>:59:59 <span class="o">(</span>UTC<span class="o">)</span> www.2fwld67ac2.com,www.6suxdq3miwwewq4.net
<span class="m">13</span>-12-18 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-02-14 <span class="m">23</span>:59:59 <span class="o">(</span>UTC<span class="o">)</span> www.npmxal2ohuefme26yf.com,www.c7kriuquvh.net
<span class="m">13</span>-10-18 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-06-16 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.s426lumoi7.com,www.ouzbot23a6lw3vvmszx.net
<span class="m">13</span>-12-31 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-02-01 <span class="m">23</span>:59:59 <span class="o">(</span>UTC<span class="o">)</span> www.vywbff5wkza6npkd5l.com,www.ugdrrog5ro5wdfddj.net
<span class="m">13</span>-11-27 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-08-13 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.ozsx22b4nda.com,www.lr7s5k3n6ber.net
<span class="m">13</span>-03-31 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-01-06 <span class="m">23</span>:59:59 <span class="o">(</span>UTC<span class="o">)</span> www.plgx26wgyroot37x3ysj.com,www.xwx5gpj5t2msq3.net
<span class="m">13</span>-12-18 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-02-20 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.gempmzrnwnk.com,www.6lrz7wtwprz.net
<span class="m">13</span>-08-16 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-01-26 <span class="m">23</span>:59:59 <span class="o">(</span>UTC<span class="o">)</span> www.rxy4jiw4wk.com,www.g66mipkcyhjwumywk4h.net
<span class="m">13</span>-12-30 <span class="m">19</span>:07:41 <span class="o">(</span>UTC<span class="o">)</span>,14-12-30 <span class="m">19</span>:07:41 <span class="o">(</span>UTC<span class="o">)</span> www.o5qzqtbs.com,www.bnymkm3nk7jtz3.net
<span class="m">13</span>-07-27 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-01-18 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.rtqtkopfct767ai.com,www.facp2b2y5wjffbo5ioy.net
<span class="m">13</span>-09-09 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-02-26 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.lvv4l6sx3qafei2s5u.com,www.vznlngjz7a2fpg.net
<span class="m">13</span>-12-21 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-02-08 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.mbrdx4tz2ob5wlvazlr.com,www.shxl35n3zt.net
<span class="m">13</span>-12-12 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-01-15 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.4jvdpoo5wcklhd3usu.com,www.f4uxyorx2h.net
<span class="m">13</span>-10-17 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-05-05 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.zcgg5yiwzajal4.com,www.55a4kx5jrqxezvk.net
<span class="m">13</span>-05-18 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-04-07 <span class="m">23</span>:59:59 <span class="o">(</span>UTC<span class="o">)</span> www.3eexfeaw.com,www.iedhzej4tie4egm.net
<span class="m">13</span>-12-23 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-01-22 <span class="m">23</span>:59:59 <span class="o">(</span>UTC<span class="o">)</span> www.5m6ywj2w7zs.com,www.iolbr3jbfs.net
<span class="m">13</span>-03-09 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-01-01 <span class="m">23</span>:59:59 <span class="o">(</span>UTC<span class="o">)</span> www.hbwpqbx4zimtptui.com,www.77wneeix55t.net
<span class="m">13</span>-12-26 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-04-19 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.pxznjv3t75.com,www.wuqq77l634eogfm.net
<span class="m">13</span>-12-07 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-03-17 <span class="m">23</span>:59:59 <span class="o">(</span>UTC<span class="o">)</span> www.6pp7bfbdywvcaicqmfq.com,www.g6oa3qdobmdgl5tprm.net
<span class="m">13</span>-12-30 <span class="m">19</span>:42:49 <span class="o">(</span>UTC<span class="o">)</span>,14-12-30 <span class="m">19</span>:42:49 <span class="o">(</span>UTC<span class="o">)</span> www.twngp3xrqgo4p.com,www.znskvp5k5pns22y2.net
<span class="m">13</span>-02-14 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span>,14-01-14 <span class="m">00</span>:00:00 <span class="o">(</span>UTC<span class="o">)</span> www.spx5a4e5eyhkdtpt2xj.com,www.6phyovjhggkfm.net
</pre></div>
<p>So with this knowledge I started looking to see if there were any current methods of identifying the anomalous certificate identifiers. Lucky for Bro users, <a href="https://github.com/sethhall/">Seth Hall</a> created a <a href="https://raw.github.com/sethhall/bro-junk-drawer/master/detect-tor.bro">detect-tor.bro</a> script to do just that. I <a href="http://www.bro.org/download/">downloaded</a> the latest Bro 2.4 source package and built it on my Ubuntu <span class="caps">VM</span>. I also pulled down the aforementioned detect-tor.bro script. I was greeted with a warning and did not see the expected logs:</p>
<div class="highlight"><pre><span></span>$ sudo /usr/local/bro/bin/bro -r tor.pcap detect-tor.bro
warning in /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line <span class="m">54</span>: Your trace file likely has invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the <span class="s1">'ignore_checksums'</span> variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted.
</pre></div>
<p>This was quickly fixed by including the <code>-C</code> toggle in order to ignore checksums.</p>
<div class="highlight"><pre><span></span>$ sudo /usr/local/bro/bin/bro -C -r tor.pcap detect-tor.bro
</pre></div>
<p>After parsing the Tor traffic collected via Wireshark or tcpdump, Bro should have generated some logs. At first glace, we see an alert from the detect-tor.bro script. While the event is pretty self explanatory, note the destination <span class="caps">IP</span> addresses are not included because Tor will usually have multiple servers, i.e. destination <span class="caps">IP</span> addresses.</p>
<div class="highlight"><pre><span></span>$ more notice.log
<span class="c1">#separator \x09</span>
<span class="c1">#set_separator ,</span>
<span class="c1">#empty_field (empty)</span>
<span class="c1">#unset_field -</span>
<span class="c1">#path notice</span>
<span class="c1">#open 2014-01-03-14-12-05</span>
<span class="c1">#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc proto note msg sub src dst p n peer_descr actions suppress_for dropped remote_location.country_code remote_location.region remote_location.city remote_locatio</span>
n.latitude remote_location.longitude
<span class="c1">#types time string addr port addr port string string string enum enum string string addr addr port count string table[enum] interval bool string string string double double</span>
<span class="m">1388434821</span>.597322 - - - - - - - - - DetectTor::Found <span class="m">10</span>.0.0.126 was found using Tor by connecting t
o servers with at least <span class="m">10</span> unique weird certs - <span class="m">10</span>.0.0.126 - - - bro Notice::ACTION_LOG <span class="m">3600</span>.000000 F - - -- -
<span class="c1">#close 2014-01-03-14-12-05</span>
</pre></div>
<p>We can cut down column noise by specifying only what we want to see:</p>
<div class="highlight"><pre><span></span>$ cat notice.log<span class="p">|</span>/usr/local/bro/bin/bro-cut -c -d note msg src dst actions suppress_for dropped
<span class="c1">#separator \x09</span>
<span class="c1">#set_separator ,</span>
<span class="c1">#empty_field (empty)</span>
<span class="c1">#unset_field -</span>
<span class="c1">#path notice</span>
<span class="c1">#open 2014-01-03-14-12-05</span>
<span class="c1">#fields note msg src dst actions suppress_for dropped</span>
<span class="c1">#types string string addr addr table[enum] interval bool</span>
DetectTor::Found <span class="m">10</span>.0.0.126 was found using Tor by connecting to servers with at least <span class="m">10</span> unique weird certs <span class="m">10</span>.0.0.126 - Notice::ACTION_LOG <span class="m">3600</span>.000000 F
</pre></div>
<p>After seeing the alert in the <code>notice.log</code>, we look in the <code>ssl.log</code> file as well in order to determine what traffic caused the alert to fire.</p>
<div class="highlight"><pre><span></span>$ more ssl.log
<span class="c1">#separator \x09</span>
<span class="c1">#set_separator ,</span>
<span class="c1">#empty_field (empty)</span>
<span class="c1">#unset_field -</span>
<span class="c1">#path ssl</span>
<span class="c1">#open 2014-01-03-14-12-05</span>
<span class="c1">#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p version cipher server_name session_id subject issuer_subject not_va</span>
lid_before not_valid_after last_alert client_subject client_issuer_subject
<span class="c1">#types time string addr port addr port string string string string string string time time string string string</span>
<span class="m">1388434821</span>.514935 CwRHlF31djcMrO7Z98 <span class="m">10</span>.0.0.126 <span class="m">51191</span> <span class="m">199</span>.36.221.196 <span class="m">9001</span> TLSv10 TLS_DHE_RSA_WITH_AES_256_CBC_SHA www.wplgkqpnteb.com -CN<span class="o">=</span>www.ri6ufvqioii5se5tzbgt.net <span class="nv">CN</span><span class="o">=</span>www.dyyp6enzivlm46.com <span class="m">1388447336</span>.000000 <span class="m">1419983336</span>.000000 - - -
<span class="m">1388434821</span>.482053 Ck1Mgy4ubChMFyneFc <span class="m">10</span>.0.0.126 <span class="m">38946</span> <span class="m">198</span>.27.97.223 <span class="m">443</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.p65b.com - <span class="nv">CN</span><span class="o">=</span>www.hkkch64skp7am.net <span class="nv">CN</span><span class="o">=</span>www.axslhtfqq.com <span class="m">1381809600</span>.000000 <span class="m">1392181199</span>.000000 - - -
<span class="m">1388434821</span>.533291 CZOEio3mxlQgpmVD2i <span class="m">10</span>.0.0.126 <span class="m">36715</span> <span class="m">149</span>.9.0.60 <span class="m">9001</span> TLSv10 TLS_DHE_RSA_WITH_AES_256_CBC_SHA www.dpvdl3n6yzwv.com -CN<span class="o">=</span>www.anojueopqlpgsj.net <span class="nv">CN</span><span class="o">=</span>www.u2rsltgpogir6t.com <span class="m">1384405200</span>.000000 <span class="m">1398830399</span>.000000 - - -
<span class="m">1388434821</span>.484476 CnU0VyJcJHaeCaxh8 <span class="m">10</span>.0.0.126 <span class="m">49341</span> <span class="m">66</span>.18.12.197 <span class="m">443</span> TLSv10 TLS_DHE_RSA_WITH_AES_256_CBC_SHA www.6kyx72vjlrwxcmxnj4
we7n.com - <span class="nv">CN</span><span class="o">=</span>www.4tdznzbrfuv.net <span class="nv">CN</span><span class="o">=</span>www.igdpzct5tauwgyqs.com <span class="m">1388446368</span>.000000 <span class="m">1419982368</span>.000000 - - -
<span class="m">1388434821</span>.484255 Cc00yR3kKWb2GstwXf <span class="m">10</span>.0.0.126 <span class="m">40742</span> <span class="m">64</span>.62.249.222 <span class="m">443</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.de5v2whiex3xxy.com
- <span class="nv">CN</span><span class="o">=</span>www.glk3fwiz6.net <span class="nv">CN</span><span class="o">=</span>www.3pzqe4en5.com <span class="m">1384664400</span>.000000 <span class="m">1403409600</span>.000000 - - -
<span class="m">1388434821</span>.583284 CuVFNK14saFKjGVhfh <span class="m">10</span>.0.0.126 <span class="m">54393</span> <span class="m">50</span>.115.122.68 <span class="m">9001</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.ojj4rbje7z7.com -CN<span class="o">=</span>www.qexiojanju56.net <span class="nv">CN</span><span class="o">=</span>www.nnfslkrseh.com <span class="m">1387342800</span>.000000 <span class="m">1390280400</span>.000000 - - -
<span class="m">1388434821</span>.482585 CROLl5Vd0jUzvvwn <span class="m">10</span>.0.0.126 <span class="m">46797</span> <span class="m">212</span>.83.140.45 <span class="m">443</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.esd7jqvwpbwebf.com
- <span class="nv">CN</span><span class="o">=</span>www.nolspqtib3ix.net <span class="nv">CN</span><span class="o">=</span>www.3pxivyds.com <span class="m">1380859200</span>.000000 <span class="m">1398139200</span>.000000 - - -
<span class="m">1388434821</span>.597288 CXemGQ4G0PFf5DvUf <span class="m">10</span>.0.0.126 <span class="m">34887</span> <span class="m">72</span>.52.91.30 <span class="m">5901</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.igyewbs5.com -CN<span class="o">=</span>www.bnlln35al.net <span class="nv">CN</span><span class="o">=</span>www.henq76fjat2ozl2537.com <span class="m">1376020800</span>.000000 <span class="m">1403841600</span>.000000 - - -
<span class="m">1388434821</span>.597322 CFrNiH22BOLl917zjl <span class="m">10</span>.0.0.126 <span class="m">56135</span> <span class="m">144</span>.76.109.178 <span class="m">9081</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.57xl.com - <span class="nv">CN</span><span class="o">=</span>www.3rvuayihf4t35h.net <span class="nv">CN</span><span class="o">=</span>www.viw7rvktu36ov.com <span class="m">1386651600</span>.000000 <span class="m">1388811600</span>.000000 - - -
<span class="m">1388434821</span>.489984 CxEp7Xmn9AOlkxn0e <span class="m">10</span>.0.0.126 <span class="m">44997</span> <span class="m">31</span>.7.186.228 <span class="m">443</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.ewrk2xtmr.com -CN<span class="o">=</span>www.orutxjqwf.net <span class="nv">CN</span><span class="o">=</span>www.5orbut4ufhohm5rlj47.com <span class="m">1371614400</span>.000000 <span class="m">1397966400</span>.000000 - - -
</pre></div>
<p>Again, we can select the fields we want to see in order to minimize output.</p>
<div class="highlight"><pre><span></span>$ cat ssl.log<span class="p">|</span>/usr/local/bro/bin/bro-cut -c -d ts uid id.orig_h id.orig_p id.resp_h id.resp_p version cipher server_name subject issuer_subject not_valid_before not_valid_after
<span class="c1">#separator \x09</span>
<span class="c1">#set_separator ,</span>
<span class="c1">#empty_field (empty)</span>
<span class="c1">#unset_field -</span>
<span class="c1">#path ssl</span>
<span class="c1">#open 2014-01-03-14-12-05</span>
<span class="c1">#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p version cipher server_name subject issuer_subject not_valid_before not_valid_after</span>
<span class="c1">#types string string addr port addr port string string string string string time string</span>
<span class="m">2013</span>-12-30T15:20:21-0500 CwRHlF31djcMrO7Z98 <span class="m">10</span>.0.0.126 <span class="m">51191</span> <span class="m">199</span>.36.221.196 <span class="m">9001</span> TLSv10 TLS_DHE_RSA_WITH_AES_256_CBC_SHA www.wplgkqpnteb.com <span class="nv">CN</span><span class="o">=</span>www.ri6ufvqioii5se5tzbgt.net <span class="nv">CN</span><span class="o">=</span>www.dyyp6enzivlm46.com <span class="m">2013</span>-12-30T18:48:56-0500 <span class="m">2014</span>-12-30T18:48:56-0500
<span class="m">2013</span>-12-30T15:20:21-0500 Ck1Mgy4ubChMFyneFc <span class="m">10</span>.0.0.126 <span class="m">38946</span> <span class="m">198</span>.27.97.223 <span class="m">443</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.p65b.com <span class="nv">CN</span><span class="o">=</span>www.hkkch64skp7am.net <span class="nv">CN</span><span class="o">=</span>www.axslhtfqq.com <span class="m">2013</span>-10-15T00:00:00-0400 <span class="m">2014</span>-02-11T23:59:59-0500
<span class="m">2013</span>-12-30T15:20:21-0500 CZOEio3mxlQgpmVD2i <span class="m">10</span>.0.0.126 <span class="m">36715</span> <span class="m">149</span>.9.0.60 <span class="m">9001</span> TLSv10 TLS_DHE_RSA_WITH_AES_256_CBC_SHA www.dpvdl3n6yzwv.com <span class="nv">CN</span><span class="o">=</span>www.anojueopqlpgsj.net <span class="nv">CN</span><span class="o">=</span>www.u2rsltgpogir6t.com <span class="m">2013</span>-11-14T00:00:00-0500 <span class="m">2014</span>-04-29T23:59:59-0400
<span class="m">2013</span>-12-30T15:20:21-0500 CnU0VyJcJHaeCaxh8 <span class="m">10</span>.0.0.126 <span class="m">49341</span> <span class="m">66</span>.18.12.197 <span class="m">443</span> TLSv10 TLS_DHE_RSA_WITH_AES_256_CBC_SHA www.6kyx72vjlrwxcmxnj4we7n.com <span class="nv">CN</span><span class="o">=</span>www.4tdznzbrfuv.net <span class="nv">CN</span><span class="o">=</span>www.igdpzct5tauwgyqs.com <span class="m">2013</span>-12-30T18:32:48-0500 <span class="m">2014</span>-12-30T18:32:48-0500
<span class="m">2013</span>-12-30T15:20:21-0500 Cc00yR3kKWb2GstwXf <span class="m">10</span>.0.0.126 <span class="m">40742</span> <span class="m">64</span>.62.249.222 <span class="m">443</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.de5v2whiex3xxy.com <span class="nv">CN</span><span class="o">=</span>www.glk3fwiz6.net <span class="nv">CN</span><span class="o">=</span>www.3pzqe4en5.com <span class="m">2013</span>-11-17T00:00:00-0500 <span class="m">2014</span>-06-22T00:00:00-0400
<span class="m">2013</span>-12-30T15:20:21-0500 CuVFNK14saFKjGVhfh <span class="m">10</span>.0.0.126 <span class="m">54393</span> <span class="m">50</span>.115.122.68 <span class="m">9001</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.ojj4rbje7z7.com <span class="nv">CN</span><span class="o">=</span>www.qexiojanju56.net <span class="nv">CN</span><span class="o">=</span>www.nnfslkrseh.com <span class="m">2013</span>-12-18T00:00:00-0500 <span class="m">2014</span>-01-21T00:00:00-0500
<span class="m">2013</span>-12-30T15:20:21-0500 CROLl5Vd0jUzvvwn <span class="m">10</span>.0.0.126 <span class="m">46797</span> <span class="m">212</span>.83.140.45 <span class="m">443</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.esd7jqvwpbwebf.com <span class="nv">CN</span><span class="o">=</span>www.nolspqtib3ix.net <span class="nv">CN</span><span class="o">=</span>www.3pxivyds.com <span class="m">2013</span>-10-04T00:00:00-0400 <span class="m">2014</span>-04-22T00:00:00-0400
<span class="m">2013</span>-12-30T15:20:21-0500 CXemGQ4G0PFf5DvUf <span class="m">10</span>.0.0.126 <span class="m">34887</span> <span class="m">72</span>.52.91.30 <span class="m">5901</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.igyewbs5.com <span class="nv">CN</span><span class="o">=</span>www.bnlln35al.net <span class="nv">CN</span><span class="o">=</span>www.henq76fjat2ozl2537.com <span class="m">2013</span>-08-09T00:00:00-0400 <span class="m">2014</span>-06-27T00:00:00-0400
<span class="m">2013</span>-12-30T15:20:21-0500 CFrNiH22BOLl917zjl <span class="m">10</span>.0.0.126 <span class="m">56135</span> <span class="m">144</span>.76.109.178 <span class="m">9081</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.57xl.com <span class="nv">CN</span><span class="o">=</span>www.3rvuayihf4t35h.net <span class="nv">CN</span><span class="o">=</span>www.viw7rvktu36ov.com <span class="m">2013</span>-12-10T00:00:00-0500 <span class="m">2014</span>-01-04T00:00:00-0500
<span class="m">2013</span>-12-30T15:20:21-0500 CxEp7Xmn9AOlkxn0e <span class="m">10</span>.0.0.126 <span class="m">44997</span> <span class="m">31</span>.7.186.228 <span class="m">443</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.ewrk2xtmr.com <span class="nv">CN</span><span class="o">=</span>www.orutxjqwf.net <span class="nv">CN</span><span class="o">=</span>www.5orbut4ufhohm5rlj47.com <span class="m">2013</span>-06-19T00:00:00-0400 <span class="m">2014</span>-04-20T00:00:00-0400
<span class="m">2013</span>-12-30T15:20:21-0500 CwzpD92UikR0USUErj <span class="m">10</span>.0.0.126 <span class="m">58912</span> <span class="m">91</span>.121.113.70 <span class="m">9001</span> TLSv10 TLS_DHE_RSA_WITH_AES_256_CBC_SHA www.dv2nzruzkuf2ncqzpxh5vpg.com <span class="nv">CN</span><span class="o">=</span>www.an2nldahkafrkz6qx.net <span class="nv">CN</span><span class="o">=</span>www.ejybbncghc3qjraztwpr.com <span class="m">2013</span>-12-30T19:35:37-0500 <span class="m">2014</span>-12-30T19:35:37-0500
<span class="m">2013</span>-12-30T15:20:21-0500 CqAdrg1JryZY3kTrZ5 <span class="m">10</span>.0.0.126 <span class="m">46649</span> <span class="m">5</span>.135.187.167 <span class="m">9001</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.3h2eyn3jwsjkggg3.com <span class="nv">CN</span><span class="o">=</span>www.mt5unawhy.net <span class="nv">CN</span><span class="o">=</span>www.nexscb2bdms.com <span class="m">2013</span>-12-16T00:00:00-0500 <span class="m">2014</span>-01-10T23:59:59-0500
<span class="m">2013</span>-12-30T15:20:21-0500 CWYgR82bEI9IjcHp7a <span class="m">10</span>.0.0.126 <span class="m">37960</span> <span class="m">212</span>.83.158.5 <span class="m">443</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.w5wtl.comCN<span class="o">=</span>www.6suxdq3miwwewq4.net <span class="nv">CN</span><span class="o">=</span>www.2fwld67ac2.com <span class="m">2013</span>-08-12T00:00:00-0400 <span class="m">2014</span>-04-16T23:59:59-0400
<span class="m">2013</span>-12-30T15:20:21-0500 CpGUEo3d5jBpzI6L04 <span class="m">10</span>.0.0.126 <span class="m">50935</span> <span class="m">212</span>.83.158.50 <span class="m">443</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.lm6zdbm5w2jd5wxtmsfpkn.com <span class="nv">CN</span><span class="o">=</span>www.ouzbot23a6lw3vvmszx.net <span class="nv">CN</span><span class="o">=</span>www.s426lumoi7.com <span class="m">2013</span>-10-18T00:00:00-0400 <span class="m">2014</span>-06-16T00:00:00-0400
<span class="m">2013</span>-12-30T15:20:21-0500 CYocU22O3RREM4dfnl <span class="m">10</span>.0.0.126 <span class="m">49609</span> <span class="m">88</span>.159.20.120 <span class="m">443</span> TLSv10 TLS_DHE_RSA_WITH_AES_256_CBC_SHA www.exr2poqlv774jn4ddyvf5vvv.com <span class="nv">CN</span><span class="o">=</span>www.qzsg2ioaoplbs2gaha5.net <span class="nv">CN</span><span class="o">=</span>www.s5rc22gpzrwt4e.com <span class="m">2013</span>-12-30T19:54:02-0500 <span class="m">2014</span>-12-30T19:54:02-0500
<span class="m">2013</span>-12-30T15:20:21-0500 CxG1gw2N7G5uvDpiD2 <span class="m">10</span>.0.0.126 <span class="m">57656</span> <span class="m">95</span>.211.225.167 <span class="m">443</span> TLSv10 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA www.mwqdszwnojnepwmw4souyw.com <span class="nv">CN</span><span class="o">=</span>www.rnspic4yus5crf6w.net <span class="nv">CN</span><span class="o">=</span>www.75ba5lymxpbhw3a2kb.com <span class="m">2013</span>-11-19T00:00:00-0500 <span class="m">2014</span>-02-05T23:59:59-0500
<span class="m">2013</span>-12-30T15:20:21-0500 CcVZHF3a5TkT9byG2e <span class="m">10</span>.0.0.126 <span class="m">60680</span> <span class="m">80</span>.100.45.156 <span class="m">443</span> TLSv10 TLS_DHE_RSA_WITH_AES_128_CBC_SHA www.emqfcc55o7a4u4ecq3w63.com <span class="nv">CN</span><span class="o">=</span>www.c7kriuquvh.net <span class="nv">CN</span><span class="o">=</span>www.npmxal2ohuefme26yf.com <span class="m">2013</span>-12-18T00:00:00-0500 <span class="m">2014</span>-02-14T23:59:59-0500
</pre></div>
<p>Pretty straight forward process to identify Tor usage on a network. This could be coupled with matching the destination addresses with the Tor server list available <a href="https://www.dan.me.uk/torlist/">servers</a> or <a href="https://exonerator.torproject.org">here</a> in order to provide further validation of Tor traffic.</p>SiLK Network Traffic Analysis Visualization with R and Rayon2015-11-07T12:00:00-05:002015-11-07T12:00:00-05:00Stephen Reesetag:www.rsreese.com,2015-11-07:/silk-network-traffic-analysis-visualization-with-r-and-rayon/In this post, the process for retroactively identifying and graphing a HTTPS DDoS of service condition is described. Why do we care about graphing, because it can be a great way to describe data to folks that may not be interested in looking at it in a tabular form, e …<p>In this post, the process for retroactively identifying and graphing a <span class="caps">HTTPS</span> DDoS of service condition is described. Why do we care about graphing, because it can be a great way to describe data to folks that may not be interested in looking at it in a tabular form, e.g. leadership. The specific example will use data collected from the server this blog is hosted on. If you are following along, this post assumes you have SiLK deployed in some manner and are collecting <span class="caps">HTTP</span> or similar traffic. Technically a DDoS condition did not occur (only two hosts were making a large number of requests) but blitz.io was used to exceed the network traffic this website typically experiences for sake of example. If a true DoS condition occurred, it would appear differently as the sensor is hosted on the same node therefore it would not record the surge in traffic. In order to record a true DoS, the sensor would ideally be placed upstream in the carrier or somewhere that exceeds the devices being monitored capacity. I would like to thank network defense analyst <a href="https://www.linkedin.com/profile/view?id=ADEAAADy7VgB1LVykcDh0APWz0yz_ROaSvn-V4A">Geoffrey Sanders</a> for providing R langauge as well as statistical recommendations in order to improve data analysis and graphical representations.</p>
<p>If we would like to retroactively search for anomaly in traffic volume, we can query a number of days and look for unusual spikes:</p>
<div class="highlight"><pre><span></span>for DAY in {1..31}; do
if [ <span class="cp">${</span><span class="n">DAY</span><span class="cp">}</span> -le 9 ]; then
DAY=0<span class="cp">${</span><span class="n">DAY</span><span class="cp">}</span>
fi
RESULT=$(rwfilter --start-date=2015/07/<span class="cp">${</span><span class="n">DAY</span><span class="cp">}</span> --end-date=2015/07/<span class="cp">${</span><span class="n">DAY</span><span class="cp">}</span> --dport=443 --pass=stdout --type=all|rwuniq --fields=dport,proto -
-values=records --no-col --no-final-del --no-title --packets=20-)
echo "2015/07/<span class="cp">${</span><span class="n">DAY</span><span class="cp">}</span>|<span class="cp">${</span><span class="n">RESULT</span><span class="cp">}</span>" >> http.out
done
</pre></div>
<p>On the 21st, we see a large number of requests that significantly exceed other days:</p>
<div class="highlight"><pre><span></span>2015/07/01|443|6|1443|33287
2015/07/02|443|6|1271|30583
2015/07/03|443|6|1776|32622
2015/07/04|443|6|1498|28316
2015/07/05|443|6|1124|34428
2015/07/06|443|6|1672|36113
2015/07/07|443|6|1298|31087
2015/07/08|443|6|1629|40990
2015/07/09|443|6|42005|750922
2015/07/10|443|6|1656|54450
2015/07/11|443|6|1464|40205
2015/07/12|443|6|1279|22251
2015/07/13|443|6|1884|40887
2015/07/14|443|6|1724|49821
2015/07/15|443|6|1635|37133
2015/07/16|443|6|1653|33433
2015/07/17|443|6|1695|37580
2015/07/18|443|6|1301|24899
2015/07/19|443|6|1445|29230
2015/07/20|443|6|1314|40543
2015/07/21|443|6|70533|817855
2015/07/22|443|6|1909|42257
2015/07/23|443|6|1462|47961
2015/07/24|443|6|1705|37581
2015/07/25|443|6|1150|27093
2015/07/26|443|6|1208|21267
2015/07/27|443|6|1597|32414
2015/07/28|443|6|1714|45208
2015/07/29|443|6|1702|35607
2015/07/30|443|6|1710|46748
2015/07/31|443|6|1514|47915
</pre></div>
<p>We can use a similar query broken down by hour for the questionable day:</p>
<div class="highlight"><pre><span></span>for HOUR in {0..23}; do if [ <span class="cp">${</span><span class="n">HOUR</span><span class="cp">}</span> -le 9 ]; then
HOUR=0<span class="cp">${</span><span class="n">HOUR</span><span class="cp">}</span>
fi
RESULT=$(rwfilter --start-date=2015/07/21:<span class="cp">${</span><span class="n">HOUR</span><span class="cp">}</span> --end-date=2015/07/21:<span class="cp">${</span><span class="n">HOUR</span><span class="cp">}</span> --dport=443 --pass=stdout --type=all|rwuniq --fields=dport
,proto --values=records --no-col --no-final-del --no-title --packets=20-)
echo "<span class="cp">${</span><span class="n">HOUR</span><span class="cp">}</span>|<span class="cp">${</span><span class="n">RESULT</span><span class="cp">}</span>" >> http-hour.out
done
</pre></div>
<p>The results from the hourly query clearly depict when the surge of <span class="caps">HTTPS</span> traffic volume occurred. From here, an analyst may run more specific queries to determine if it is indeed a distributed attack or sourced from only a few nodes.</p>
<div class="highlight"><pre><span></span>00|443|6|72|1392
01|443|6|48|3203
02|443|6|151|2605
03|443|6|173|1612
04|443|6|125|2318
05|443|6|149|2622
06|443|6|72|1450
07|443|6|71|1294
08|443|6|73|1524
09|443|6|76|1881
10|443|6|67|1412
11|443|6|823|6720
12|443|6|60|1639
13|443|6|65|1511
14|443|6|72|2987
15|443|6|121|2061
16|443|6|69|2135
17|443|6|67562|722727
18|443|6|203|3222
19|443|6|112|4004
20|443|6|99|1526
21|443|6|122|44746
22|443|6|94|2129
23|443|6|54|1135
</pre></div>
<p>We can represent the tabular data from the daily and hourly queries using <a href="http://www.r-project.org">R Project</a> and ggplot2. For the daily plot example, add a header <code>day|dPort|protocol|Records|Packets</code> to the dataset and run <code>Rscript filename.r dataset.dat</code> replacing the command directives with the script below and your dataset:</p>
<div class="highlight"><pre><span></span><span class="kn">library</span><span class="p">(</span>ggplot2<span class="p">)</span>
<span class="kn">library</span><span class="p">(</span>reshape2<span class="p">)</span>
<span class="kp">options</span><span class="p">(</span><span class="s">"scipen"</span><span class="o">=</span><span class="m">100</span><span class="p">,</span> <span class="s">"digits"</span><span class="o">=</span><span class="m">4</span><span class="p">)</span>
fname <span class="o"><-</span> <span class="kp">commandArgs</span><span class="p">(</span>trailingOnly <span class="o">=</span> <span class="kc">TRUE</span><span class="p">)[</span><span class="m">1</span><span class="p">]</span>
flowrecs <span class="o"><-</span> read.table<span class="p">(</span>fname<span class="p">,</span> header <span class="o">=</span> <span class="kc">TRUE</span><span class="p">,</span> sep <span class="o">=</span> <span class="s">"|"</span><span class="p">)</span>
flowrecs<span class="o">$</span>day <span class="o"><-</span> <span class="kp">as.Date</span><span class="p">(</span>flowrecs<span class="o">$</span>day<span class="p">,</span> <span class="s">"%Y/%m/%d"</span><span class="p">)</span>
test_data_long <span class="o"><-</span> melt<span class="p">(</span>flowrecs<span class="p">,</span> id.vars<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="s">"day"</span><span class="p">,</span> <span class="s">"dPort"</span><span class="p">,</span> <span class="s">"protocol"</span><span class="p">))</span>
flow.plot <span class="o"><-</span> ggplot<span class="p">(</span>data<span class="o">=</span>test_data_long<span class="p">,</span>
aes<span class="p">(</span>x<span class="o">=</span>day<span class="p">,</span> y<span class="o">=</span>value<span class="p">,</span> colour<span class="o">=</span>variable<span class="p">))</span>
geom_line<span class="p">()</span> <span class="o">+</span> geom_point<span class="p">()</span> <span class="o">+</span> xlab<span class="p">(</span><span class="s">"Day"</span><span class="p">)</span> <span class="o">+</span> ylab<span class="p">(</span><span class="s">"Flow Records with 20+ Packets"</span><span class="p">)</span>
ggtitle<span class="p">(</span><span class="kp">paste</span><span class="p">(</span><span class="s">"Flow Records by Destination Port"</span><span class="p">))</span>
png<span class="p">(</span><span class="s">"plot.png"</span><span class="p">,</span> width<span class="o">=</span><span class="m">1200</span><span class="p">,</span> height<span class="o">=</span><span class="m">400</span><span class="p">)</span>
plot<span class="p">(</span>flow.plot<span class="p">)</span>
dev.off<span class="p">()</span>
</pre></div>
<p>Which should provide a graphical representation similar to:</p>
<p><a href="https://www.rsreese.com/assets/daily-https-plot.png"><img alt="daily-https-plot" src="https://www.rsreese.com/assets/daily-https-plot.png"></a></p>
<p>Similarly, we can do the same thing with the hourly plot by specifying the correct header of <code>hour|dPort|protocol|Records|Packets</code> and rerunning <code>Rscript</code> in the same manner as the daily plot.</p>
<div class="highlight"><pre><span></span><span class="kn">library</span><span class="p">(</span>ggplot2<span class="p">)</span>
<span class="kn">library</span><span class="p">(</span>reshape2<span class="p">)</span>
<span class="kp">options</span><span class="p">(</span><span class="s">"scipen"</span><span class="o">=</span><span class="m">100</span><span class="p">,</span> <span class="s">"digits"</span><span class="o">=</span><span class="m">4</span><span class="p">)</span>
fname <span class="o"><-</span> <span class="kp">commandArgs</span><span class="p">(</span>trailingOnly <span class="o">=</span> <span class="kc">TRUE</span><span class="p">)[</span><span class="m">1</span><span class="p">]</span>
flowrecs <span class="o"><-</span> read.table<span class="p">(</span>fname<span class="p">,</span> header <span class="o">=</span> <span class="kc">TRUE</span><span class="p">,</span> sep <span class="o">=</span> <span class="s">"|"</span><span class="p">)</span>
flowrecs<span class="o">$</span>hour <span class="o"><-</span> <span class="kp">factor</span><span class="p">(</span>flowrecs<span class="o">$</span>hour<span class="p">,</span> levels<span class="o">=</span><span class="kp">unique</span><span class="p">(</span>flowrecs<span class="o">$</span>hour<span class="p">))</span>
test_data_long <span class="o"><-</span> melt<span class="p">(</span>flowrecs<span class="p">,</span> id.var<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="s">"hour"</span><span class="p">,</span> <span class="s">"dPort"</span><span class="p">,</span> <span class="s">"protocol"</span><span class="p">))</span>
flow.plot <span class="o"><-</span> ggplot<span class="p">(</span>data<span class="o">=</span>test_data_long<span class="p">,</span>
aes<span class="p">(</span>x<span class="o">=</span>hour<span class="p">,</span> y<span class="o">=</span>value<span class="p">,</span> colour<span class="o">=</span>variable<span class="p">,</span> group<span class="o">=</span>variable<span class="p">))</span> <span class="o">+</span>
geom_line<span class="p">()</span> <span class="o">+</span> geom_point<span class="p">()</span> <span class="o">+</span> xlab<span class="p">(</span><span class="s">"Hour"</span><span class="p">)</span> <span class="o">+</span> ylab<span class="p">(</span><span class="s">"Flow Records with 20+ Packets"</span><span class="p">)</span>
ggtitle<span class="p">(</span><span class="kp">paste</span><span class="p">(</span><span class="s">"Flow Records by Destination Port"</span><span class="p">))</span>
png<span class="p">(</span><span class="s">"plot.png"</span><span class="p">,</span> width<span class="o">=</span><span class="m">1200</span><span class="p">,</span> height<span class="o">=</span><span class="m">400</span><span class="p">)</span>
plot<span class="p">(</span>flow.plot<span class="p">)</span>
dev.off<span class="p">()</span>
</pre></div>
<p>This depicts the two large sets of requests we had in the single day:</p>
<p><a href="https://www.rsreese.com/assets/hourly-https-plot.png"><img alt="hourly-https-plot" src="https://www.rsreese.com/assets/hourly-https-plot.png"></a></p>
<p>We can use <code>rwstats</code> in order to take a look at our top talkers if we are aware of congestion or other signs that the uniformity of visitors has changed. This query is a little artificial though. It is very possible that the source attacks may come from hundreds or even thousands of bots or some reflection mechanism depending on the service. If that is the case, we may have to look at other tuples or the actual request in order to determine a similarity between the distributed attack sources.</p>
<div class="highlight"><pre><span></span>$ rwfilter --start-date<span class="o">=</span><span class="m">2015</span>/7/21 --end-date<span class="o">=</span><span class="m">2015</span>/7/21 --dport<span class="o">=</span><span class="m">443</span> --pass<span class="o">=</span>stdout --type<span class="o">=</span>all<span class="p">|</span>rwstats --fields<span class="o">=</span>sip --count<span class="o">=</span><span class="m">10</span> --no-col --no-final-del
INPUT: <span class="m">70533</span> Records <span class="k">for</span> <span class="m">551</span> Bins and <span class="m">70533</span> Total Records
OUTPUT: Top <span class="m">10</span> Bins by Records
sIP<span class="p">|</span>Records<span class="p">|</span>%Records<span class="p">|</span>cumul_%
<span class="m">54</span>.173.173.209<span class="p">|</span><span class="m">33704</span><span class="p">|</span><span class="m">47</span>.784725<span class="p">|</span><span class="m">47</span>.784725
<span class="m">54</span>.86.98.210<span class="p">|</span><span class="m">33698</span><span class="p">|</span><span class="m">47</span>.776218<span class="p">|</span><span class="m">95</span>.560943
<span class="m">162</span>.243.196.54<span class="p">|</span><span class="m">742</span><span class="p">|</span><span class="m">1</span>.051990<span class="p">|</span><span class="m">96</span>.612933
<span class="m">180</span>.76.15.142<span class="p">|</span><span class="m">97</span><span class="p">|</span><span class="m">0</span>.137524<span class="p">|</span><span class="m">96</span>.750457
<span class="m">68</span>.180.230.230<span class="p">|</span><span class="m">75</span><span class="p">|</span><span class="m">0</span>.106333<span class="p">|</span><span class="m">96</span>.856790
<span class="m">180</span>.76.15.140<span class="p">|</span><span class="m">37</span><span class="p">|</span><span class="m">0</span>.052458<span class="p">|</span><span class="m">96</span>.909248
<span class="m">72</span>.80.60.139<span class="p">|</span><span class="m">31</span><span class="p">|</span><span class="m">0</span>.043951<span class="p">|</span><span class="m">96</span>.953199
<span class="m">66</span>.249.67.118<span class="p">|</span><span class="m">31</span><span class="p">|</span><span class="m">0</span>.043951<span class="p">|</span><span class="m">96</span>.997150
<span class="m">180</span>.76.15.136<span class="p">|</span><span class="m">24</span><span class="p">|</span><span class="m">0</span>.034027<span class="p">|</span><span class="m">97</span>.031177
<span class="m">63</span>.254.26.10<span class="p">|</span><span class="m">23</span><span class="p">|</span><span class="m">0</span>.032609<span class="p">|</span><span class="m">97</span>.063786
</pre></div>
<p>We can append <code>rwresolve</code> in order resolve a specific <span class="caps">IP</span> field. We see two Amazon hosts whom are likely the Blitz.io bots as they comprise 96% of the traffic for the defined time threshold:</p>
<div class="highlight"><pre><span></span>$ rwfilter --start-date<span class="o">=</span><span class="m">2015</span>/7/21 --end-date<span class="o">=</span><span class="m">2015</span>/7/21 --dport<span class="o">=</span><span class="m">443</span> --pass<span class="o">=</span>stdout --type<span class="o">=</span>all<span class="p">|</span>rwstats --fields<span class="o">=</span>sip --count<span class="o">=</span><span class="m">10</span> --no-col --no-final-del<span class="p">|</span>rwresolve --ip-fields<span class="o">=</span><span class="m">1</span>
INPUT: <span class="m">70533</span> Records <span class="k">for</span> <span class="m">551</span> Bins and <span class="m">70533</span> Total Records
OUTPUT: Top <span class="m">10</span> Bins by Records
sIP<span class="p">|</span>Records<span class="p">|</span>%Records<span class="p">|</span>cumul_%
ec2-54-173-173-209.compute-1.amazonaws.com<span class="p">|</span><span class="m">33704</span><span class="p">|</span><span class="m">47</span>.784725<span class="p">|</span><span class="m">47</span>.784725
ec2-54-86-98-210.compute-1.amazonaws.com<span class="p">|</span><span class="m">33698</span><span class="p">|</span><span class="m">47</span>.776218<span class="p">|</span><span class="m">95</span>.560943
<span class="m">162</span>.243.196.54<span class="p">|</span><span class="m">742</span><span class="p">|</span><span class="m">1</span>.051990<span class="p">|</span><span class="m">96</span>.612933
baiduspider-180-76-15-142.crawl.baidu.com<span class="p">|</span><span class="m">97</span><span class="p">|</span><span class="m">0</span>.137524<span class="p">|</span><span class="m">96</span>.750457
b115504.yse.yahoo.net<span class="p">|</span><span class="m">75</span><span class="p">|</span><span class="m">0</span>.106333<span class="p">|</span><span class="m">96</span>.856790
baiduspider-180-76-15-140.crawl.baidu.com<span class="p">|</span><span class="m">37</span><span class="p">|</span><span class="m">0</span>.052458<span class="p">|</span><span class="m">96</span>.909248
pool-72-80-60-139.nycmny.fios.verizon.net<span class="p">|</span><span class="m">31</span><span class="p">|</span><span class="m">0</span>.043951<span class="p">|</span><span class="m">96</span>.953199
crawl-66-249-67-118.googlebot.com<span class="p">|</span><span class="m">31</span><span class="p">|</span><span class="m">0</span>.043951<span class="p">|</span><span class="m">96</span>.997150
baiduspider-180-76-15-136.crawl.baidu.com<span class="p">|</span><span class="m">24</span><span class="p">|</span><span class="m">0</span>.034027<span class="p">|</span><span class="m">97</span>.031177
mail.oswaldcompanies.com<span class="p">|</span><span class="m">23</span><span class="p">|</span><span class="m">0</span>.032609<span class="p">|</span><span class="m">97</span>.063786
</pre></div>
<p>Both of the graphs above describe the anomalous traffic but our normal traffic is no longer clear. One way we can provide a more clarification is to use statistics in order to more effectively describe the data because of the significant outliers. In order to achieve this, we will first use a <code>log</code> function in order to describe the data volumes. We use the same scripts as earlier, but change <code>y=value</code> to <code>y=log(value)</code>.</p>
<p><a href="https://www.rsreese.com/assets/daily-https-logplot.png"><img alt="daily-https-logplot" src="https://www.rsreese.com/assets/daily-https-logplot.png"></a></p>
<p><a href="https://www.rsreese.com/assets/hourly-https-logplot.png"><img alt="hourly-https-logplot" src="https://www.rsreese.com/assets/hourly-https-logplot.png"></a></p>
<p>While using a <code>log</code> function provided an improvement, it may not provide an accurate representation of volume data types. Next, we will take a look at percentiles with R. Our data frame is composed of three days. The middle being the 21st which contains our fictitious DoS attack.</p>
<div class="highlight"><pre><span></span>> mydata
[,1] [,2] [,3]
[1,] 1252 1392 1551
[2,] 1347 3203 1969
[3,] 749 2605 1642
[4,] 2232 1612 1432
[5,] 707 2318 1531
[6,] 552 2622 1175
[7,] 1072 1450 1981
[8,] 487 1294 1606
[9,] 1448 1524 959
[10,] 867 1881 1763
[11,] 903 1412 1283
[12,] 911 6720 3055
[13,] 1125 1639 3609
[14,] 1511 1511 1977
[15,] 1792 2987 2476
[16,] 912 2061 1722
[17,] 1114 2135 655
[18,] 424 722727 1338
[19,] 3888 3222 4038
[20,] 3646 4004 1765
[21,] 9281 1526 1650
[22,] 1590 44746 1190
[23,] 1131 2129 1190
[24,] 1602 1135 700
</pre></div>
<p>Here is how we get to our graph in the R console:</p>
<div class="highlight"><pre><span></span>mydata <span class="o"><-</span> <span class="kt">matrix</span><span class="p">(</span>ncol<span class="o">=</span><span class="m">24</span><span class="p">,</span> nrow<span class="o">=</span><span class="m">3</span><span class="p">)</span>
mydata <span class="o"><-</span> <span class="kt">matrix</span><span class="p">(</span>df<span class="o">$</span>Packets<span class="p">,</span> ncol<span class="o">=</span><span class="m">3</span><span class="p">,</span> nrow<span class="o">=</span><span class="m">24</span><span class="p">)</span>
dataout <span class="o"><-</span> <span class="kp">apply</span><span class="p">(</span>mydata<span class="p">,</span> <span class="m">1</span><span class="p">,</span> quantile<span class="p">,</span> probs<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">0.05</span><span class="p">,</span> <span class="m">0.5</span><span class="p">,</span> <span class="m">0.90</span><span class="p">))</span>
ylim<span class="o">=</span><span class="kp">range</span><span class="p">(</span><span class="m">500</span><span class="p">,</span><span class="m">5000</span><span class="p">)</span>
plot<span class="p">(</span><span class="kp">seq</span><span class="p">(</span><span class="kp">ncol</span><span class="p">(</span>dataout<span class="p">)),</span> dataout<span class="p">[</span><span class="m">1</span><span class="p">,],</span> t<span class="o">=</span><span class="s">"l"</span><span class="p">,</span> lty<span class="o">=</span><span class="m">2</span><span class="p">,</span> ylim<span class="o">=</span>ylim<span class="p">,</span> main<span class="o">=</span><span class="s">"Flow Percentiles"</span><span class="p">,</span> xlab<span class="o">=</span><span class="s">"Hour"</span><span class="p">,</span>
ylab<span class="o">=</span><span class="s">"Packets"</span><span class="p">)</span> <span class="c1">#5%</span>
lines<span class="p">(</span><span class="kp">seq</span><span class="p">(</span><span class="kp">ncol</span><span class="p">(</span>dataout<span class="p">)),</span> dataout<span class="p">[</span><span class="m">2</span><span class="p">,],</span> lty<span class="o">=</span><span class="m">1</span><span class="p">,</span> lwd<span class="o">=</span><span class="m">2</span><span class="p">)</span> <span class="c1">#50%</span>
lines<span class="p">(</span><span class="kp">seq</span><span class="p">(</span><span class="kp">ncol</span><span class="p">(</span>dataout<span class="p">)),</span> dataout<span class="p">[</span><span class="m">3</span><span class="p">,],</span> lty<span class="o">=</span><span class="m">2</span><span class="p">,</span> col<span class="o">=</span><span class="m">2</span><span class="p">)</span> <span class="c1">#90%</span>
legend<span class="p">(</span><span class="s">"topleft"</span><span class="p">,</span> legend<span class="o">=</span><span class="kp">rev</span><span class="p">(</span><span class="kp">rownames</span><span class="p">(</span>dataout<span class="p">)),</span> lwd<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">2</span><span class="p">,</span><span class="m">1</span><span class="p">),</span> col<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">2</span><span class="p">,</span><span class="m">1</span><span class="p">,</span><span class="m">1</span><span class="p">),</span> lty<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">2</span><span class="p">,</span><span class="m">1</span><span class="p">,</span><span class="m">2</span><span class="p">))</span>
</pre></div>
<p><a href="https://www.rsreese.com/assets/hourly-https-quantileplot.png"><img alt="hourly-https-quantileplot" src="https://www.rsreese.com/assets/hourly-https-quantileplot.png"></a></p>
<p>The 90th percentile really stands out here but having to use <code>y</code> limit to see our lower percentiles prevents us from seeing the whole picture. Let us graph both again but splitting our lower and upper bounds.</p>
<div class="highlight"><pre><span></span>mydata <span class="o"><-</span> <span class="kt">matrix</span><span class="p">(</span>ncol<span class="o">=</span><span class="m">24</span><span class="p">,</span> nrow<span class="o">=</span><span class="m">3</span><span class="p">)</span>
mydata <span class="o"><-</span> <span class="kt">matrix</span><span class="p">(</span>df<span class="o">$</span>Packets<span class="p">,</span> ncol<span class="o">=</span><span class="m">3</span><span class="p">,</span> nrow<span class="o">=</span><span class="m">24</span><span class="p">)</span>
dataout <span class="o"><-</span> <span class="kp">apply</span><span class="p">(</span>mydata<span class="p">,</span> <span class="m">1</span><span class="p">,</span> quantile<span class="p">,</span> probs<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">0.01</span><span class="p">,</span> <span class="m">0.05</span><span class="p">,</span> <span class="m">0.5</span><span class="p">))</span>
ylim<span class="o">=</span><span class="kp">range</span><span class="p">(</span><span class="m">100</span><span class="p">,</span><span class="m">4200</span><span class="p">)</span>
plot<span class="p">(</span><span class="kp">seq</span><span class="p">(</span><span class="kp">ncol</span><span class="p">(</span>dataout<span class="p">)),</span> dataout<span class="p">[</span><span class="m">1</span><span class="p">,],</span> t<span class="o">=</span><span class="s">"l"</span><span class="p">,</span> lty<span class="o">=</span><span class="m">2</span><span class="p">,</span> ylim<span class="o">=</span>ylim<span class="p">,</span> main<span class="o">=</span><span class="s">"Flow Percentiles"</span><span class="p">,</span> xlab<span class="o">=</span><span class="s">"Hour"</span><span class="p">,</span> ylab<span class="o">=</span><span class="s">"Packets"</span><span class="p">)</span> <span class="c1">#1%</span>
lines<span class="p">(</span><span class="kp">seq</span><span class="p">(</span><span class="kp">ncol</span><span class="p">(</span>dataout<span class="p">)),</span> dataout<span class="p">[</span><span class="m">2</span><span class="p">,],</span> lty<span class="o">=</span><span class="m">1</span><span class="p">,</span> lwd<span class="o">=</span><span class="m">2</span><span class="p">)</span> <span class="c1">#5%</span>
lines<span class="p">(</span><span class="kp">seq</span><span class="p">(</span><span class="kp">ncol</span><span class="p">(</span>dataout<span class="p">)),</span> dataout<span class="p">[</span><span class="m">3</span><span class="p">,],</span> lty<span class="o">=</span><span class="m">2</span><span class="p">,</span> col<span class="o">=</span><span class="m">2</span><span class="p">)</span> <span class="c1">#50%</span>
legend<span class="p">(</span><span class="s">"topleft"</span><span class="p">,</span> legend<span class="o">=</span><span class="kp">rev</span><span class="p">(</span><span class="kp">rownames</span><span class="p">(</span>dataout<span class="p">)),</span> lwd<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">2</span><span class="p">,</span><span class="m">1</span><span class="p">),</span> col<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">2</span><span class="p">,</span><span class="m">1</span><span class="p">,</span><span class="m">1</span><span class="p">),</span> lty<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">2</span><span class="p">,</span><span class="m">1</span><span class="p">,</span><span class="m">2</span><span class="p">))</span>
</pre></div>
<p><a href="https://www.rsreese.com/assets/hourly-https-lowerplot.png"><img alt="hourly-https-lowerplot" src="https://www.rsreese.com/assets/hourly-https-lowerplot.png"></a></p>
<div class="highlight"><pre><span></span>mydata <span class="o"><-</span> <span class="kt">matrix</span><span class="p">(</span>ncol<span class="o">=</span><span class="m">24</span><span class="p">,</span> nrow<span class="o">=</span><span class="m">3</span><span class="p">)</span>
mydata <span class="o"><-</span> <span class="kt">matrix</span><span class="p">(</span>df<span class="o">$</span>Packets<span class="p">,</span> ncol<span class="o">=</span><span class="m">3</span><span class="p">,</span> nrow<span class="o">=</span><span class="m">24</span><span class="p">)</span>
dataout <span class="o"><-</span> <span class="kp">apply</span><span class="p">(</span>mydata<span class="p">,</span> <span class="m">1</span><span class="p">,</span> quantile<span class="p">,</span> probs<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">0.90</span><span class="p">,</span> <span class="m">0.95</span><span class="p">))</span>
ylim<span class="o">=</span><span class="kp">range</span><span class="p">(</span>dataout<span class="p">)</span>
plot<span class="p">(</span><span class="kp">seq</span><span class="p">(</span><span class="kp">ncol</span><span class="p">(</span>dataout<span class="p">)),</span> dataout<span class="p">[</span><span class="m">1</span><span class="p">,],</span> t<span class="o">=</span><span class="s">"l"</span><span class="p">,</span> lty<span class="o">=</span><span class="m">2</span><span class="p">,</span> ylim<span class="o">=</span>ylim<span class="p">,</span> main<span class="o">=</span><span class="s">"Flow Percentiles"</span><span class="p">,</span> xlab<span class="o">=</span><span class="s">"Hour"</span><span class="p">,</span> ylab<span class="o">=</span><span class="s">"Packets"</span><span class="p">)</span> <span class="c1">#90%</span>
lines<span class="p">(</span><span class="kp">seq</span><span class="p">(</span><span class="kp">ncol</span><span class="p">(</span>dataout<span class="p">)),</span> dataout<span class="p">[</span><span class="m">2</span><span class="p">,],</span> lty<span class="o">=</span><span class="m">1</span><span class="p">,</span> lwd<span class="o">=</span><span class="m">2</span><span class="p">)</span> <span class="c1">#95%</span>
legend<span class="p">(</span><span class="s">"topleft"</span><span class="p">,</span> legend<span class="o">=</span><span class="kp">rev</span><span class="p">(</span><span class="kp">rownames</span><span class="p">(</span>dataout<span class="p">)),</span> lwd<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">2</span><span class="p">,</span><span class="m">1</span><span class="p">),</span> col<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">2</span><span class="p">,</span><span class="m">1</span><span class="p">,</span><span class="m">1</span><span class="p">),</span> lty<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">2</span><span class="p">,</span><span class="m">1</span><span class="p">,</span><span class="m">2</span><span class="p">))</span>
</pre></div>
<p><a href="https://www.rsreese.com/assets/hourly-https-upperplot.png"><img alt="hourly-https-upperplot" src="https://www.rsreese.com/assets/hourly-https-upperplot.png"></a></p>
<p>This is a better analytical view. We can infer that if we see traffic at the 90 percentile, likely something is off. For the heck of it, let us see how the percentiles compare to the mean and median, the former not being necessary as we already included it in the percentile two examples above but nevertheless.</p>
<div class="highlight"><pre><span></span>ylim<span class="o">=</span><span class="kp">range</span><span class="p">(</span><span class="m">500</span><span class="p">,</span><span class="m">5000</span><span class="p">)</span>
mydata <span class="o"><-</span> <span class="kt">matrix</span><span class="p">(</span>ncol<span class="o">=</span><span class="m">24</span><span class="p">,</span> nrow<span class="o">=</span><span class="m">3</span><span class="p">)</span>
mydata <span class="o"><-</span> <span class="kt">matrix</span><span class="p">(</span>df<span class="o">$</span>Packets<span class="p">,</span> ncol<span class="o">=</span><span class="m">3</span><span class="p">,</span> nrow<span class="o">=</span><span class="m">24</span><span class="p">)</span>
meandata <span class="o"><-</span> <span class="kp">apply</span><span class="p">(</span>mydata<span class="p">,</span> <span class="m">1</span><span class="p">,</span> <span class="kp">mean</span><span class="p">)</span>
mediandata <span class="o"><-</span> <span class="kp">apply</span><span class="p">(</span>mydata<span class="p">,</span> <span class="m">1</span><span class="p">,</span> median<span class="p">)</span>
plot<span class="p">(</span>meandata<span class="p">,</span> t<span class="o">=</span><span class="s">"l"</span><span class="p">,</span> lty<span class="o">=</span><span class="m">2</span><span class="p">,</span> ylim<span class="o">=</span>ylim<span class="p">,</span> main<span class="o">=</span><span class="s">"Flows"</span><span class="p">,</span> xlab<span class="o">=</span><span class="s">"Hour"</span><span class="p">,</span> ylab<span class="o">=</span><span class="s">"Packets"</span><span class="p">)</span>
lines<span class="p">(</span>mediandata<span class="p">,</span> lty<span class="o">=</span><span class="m">1</span><span class="p">,</span> lwd<span class="o">=</span><span class="m">2</span><span class="p">)</span>
legend<span class="p">(</span><span class="s">"topleft"</span><span class="p">,</span> legend<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="s">"Mean"</span><span class="p">,</span> <span class="s">"Median"</span><span class="p">),</span> lwd<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">2</span><span class="p">),</span> col<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">1</span><span class="p">),</span> lty<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="m">2</span><span class="p">,</span><span class="m">1</span><span class="p">))</span>
</pre></div>
<p>The tabular data outliers are obvious but we will graph it. Based on this, we could leverage around 5 to 10 percentile for normal traffic but much larger sampling would need to take place as we only used three days.</p>
<div class="highlight"><pre><span></span>> meandata
[1] 1398.333 2173.000 1665.333 1758.667 1518.667 1449.667
[7] 1501.000 1129.000 1310.333 1503.667 1199.333 3562.000
[13] 2124.333 1666.333 2418.333 1565.000 1301.333 241496.333
[19] 3716.000 3138.333 4152.333 15842.000 1483.333 1145.667
> mediandata
[1] 1392 1969 1642 1612 1531 1175 1450 1294 1448 1763 1283 3055 1639 1511 2476
[16] 1722 1114 1338 3888 3646 1650 1590 1190 1135
</pre></div>
<p><a href="https://www.rsreese.com/assets/hourly-https-meanplot.png"><img alt="hourly-https-meanplot" src="https://www.rsreese.com/assets/hourly-https-meanplot.png"></a></p>
<p>Last but not least, we are going to take a look at a tool the <a href="https://tools.netsa.cert.org/index.html">NetSA</a> team has developed for graphically representing data named <a href="https://tools.netsa.cert.org/rayon/index.html">Rayon</a>. I will provide a quick reference, but you can find more details <a href="https://resources.sei.cmu.edu/asset_files/Poster/2014_020_001_300465.pdf">here</a> and <a href="https://tools.netsa.cert.org/rayon/doc/man-rytimeseries.html">here</a>. As with R language, the outliers distort the graph so we use <code>log</code> functions for the second and third graphs in order to minimize the outlier effects. First, grab the data we are interested in:</p>
<div class="highlight"><pre><span></span>$ rwfilter --start-date<span class="o">=</span><span class="m">2015</span>/7/21 --end-date<span class="o">=</span><span class="m">2015</span>/7/21 --dport<span class="o">=</span><span class="m">443</span> --proto<span class="o">=</span><span class="m">6</span> --type<span class="o">=</span>inweb --pass<span class="o">=</span>httpsin.bin
$ rwfilter --start-date<span class="o">=</span><span class="m">2015</span>/7/21 --end-date<span class="o">=</span><span class="m">2015</span>/7/21 --dport<span class="o">=</span><span class="m">443</span> --proto<span class="o">=</span><span class="m">6</span> --type<span class="o">=</span>outweb --pass<span class="o">=</span>httpsout.bin
</pre></div>
<p>Next, export the values we need, we provide a snippet of our data:</p>
<div class="highlight"><pre><span></span>$ rwcount --bin-size<span class="o">=</span><span class="m">300</span> --no-titles --delimited httpsin.bin<span class="p">|</span>awk -F<span class="se">\|</span> <span class="s1">'{printf("%s|%s|in\n", $1, $3)}'</span> > <span class="m">2</span>-top.txt
--snip--
<span class="m">2015</span>/07/21T00:00:00<span class="p">|</span><span class="m">5003</span>.00<span class="p">|</span>in
<span class="m">2015</span>/07/21T00:05:00<span class="p">|</span><span class="m">16677</span>.47<span class="p">|</span>in
<span class="m">2015</span>/07/21T00:10:00<span class="p">|</span><span class="m">4814</span>.53<span class="p">|</span>in
<span class="m">2015</span>/07/21T00:15:00<span class="p">|</span><span class="m">4951</span>.00<span class="p">|</span>in
<span class="m">2015</span>/07/21T00:20:00<span class="p">|</span><span class="m">1440</span>.00<span class="p">|</span>in
<span class="m">2015</span>/07/21T00:25:00<span class="p">|</span><span class="m">10055</span>.00<span class="p">|</span>in
<span class="m">2015</span>/07/21T00:30:00<span class="p">|</span><span class="m">5410</span>.06<span class="p">|</span>in
<span class="m">2015</span>/07/21T00:35:00<span class="p">|</span><span class="m">1356</span>.94<span class="p">|</span>in
<span class="m">2015</span>/07/21T00:40:00<span class="p">|</span><span class="m">4346</span>.32<span class="p">|</span>in
<span class="m">2015</span>/07/21T00:45:00<span class="p">|</span><span class="m">10125</span>.04<span class="p">|</span>in
<span class="m">2015</span>/07/21T00:50:00<span class="p">|</span><span class="m">7178</span>.64<span class="p">|</span>in
<span class="m">2015</span>/07/21T00:55:00<span class="p">|</span><span class="m">16766</span>.00<span class="p">|</span>in
--snip--
$ rwcount --bin-size<span class="o">=</span><span class="m">300</span> --no-titles --delimited httpsout.bin<span class="p">|</span>awk -F<span class="se">\|</span> <span class="s1">'{printf("%s|%s|out\n", $1, $3)}'</span> > <span class="m">2</span>-btm.txt
--snip--
<span class="m">2015</span>/07/21T00:00:00<span class="p">|</span><span class="m">60615</span>.00<span class="p">|</span>out
<span class="m">2015</span>/07/21T00:05:00<span class="p">|</span><span class="m">317387</span>.87<span class="p">|</span>out
<span class="m">2015</span>/07/21T00:10:00<span class="p">|</span><span class="m">214138</span>.13<span class="p">|</span>out
<span class="m">2015</span>/07/21T00:15:00<span class="p">|</span><span class="m">60527</span>.00<span class="p">|</span>out
<span class="m">2015</span>/07/21T00:20:00<span class="p">|</span><span class="m">3500</span>.00<span class="p">|</span>out
<span class="m">2015</span>/07/21T00:25:00<span class="p">|</span><span class="m">76385</span>.00<span class="p">|</span>out
<span class="m">2015</span>/07/21T00:30:00<span class="p">|</span><span class="m">77113</span>.44<span class="p">|</span>out
<span class="m">2015</span>/07/21T00:35:00<span class="p">|</span><span class="m">32326</span>.56<span class="p">|</span>out
<span class="m">2015</span>/07/21T00:40:00<span class="p">|</span><span class="m">39375</span>.58<span class="p">|</span>out
<span class="m">2015</span>/07/21T00:45:00<span class="p">|</span><span class="m">96888</span>.67<span class="p">|</span>out
<span class="m">2015</span>/07/21T00:50:00<span class="p">|</span><span class="m">30598</span>.75<span class="p">|</span>out
<span class="m">2015</span>/07/21T00:55:00<span class="p">|</span><span class="m">313460</span>.00<span class="p">|</span>out
--snip--
</pre></div>
<p>We graph the values with <code>rwtimeseries</code>. As expected, the incoming traffic is less than the outgoing <span class="caps">HTTPS</span> response. We adjusted the scale of the second and third graph using <code>log</code>, and the last Rayon graph describes data between the 95 and 100 percentiles.</p>
<div class="highlight"><pre><span></span>$ cat <span class="m">2</span>-top.txt <span class="m">2</span>-btm.txt <span class="p">|</span> rytimeseries --style<span class="o">=</span>filled_lines --output-path<span class="o">=</span><span class="m">2</span>.png --top-filter<span class="o">=</span><span class="s2">"[2]==in"</span> --bottom-filter<span class="o">=</span><span class="s2">"[2]==out"</span> --top-column<span class="o">=</span><span class="m">1</span> --bottom-column<span class="o">=</span><span class="m">1</span> --annotate-max --value-tick-label-format<span class="o">=</span>metric --value-units<span class="o">=</span>B --title<span class="o">=</span><span class="s2">"Traffic to/from Web Servers"</span> --value-scale<span class="o">=</span>linear
</pre></div>
<p><a href="https://www.rsreese.com/assets/linear-rayon-graph.png"><img alt="linear-rayon-graph" src="https://www.rsreese.com/assets/linear-rayon-graph.png"></a></p>
<div class="highlight"><pre><span></span>$ cat <span class="m">2</span>-top.txt <span class="m">2</span>-btm.txt <span class="p">|</span> rytimeseries --style<span class="o">=</span>filled_lines --output-path<span class="o">=</span><span class="m">2</span>.png --top-filter<span class="o">=</span><span class="s2">"[2]==in"</span> --bottom-filter<span class="o">=</span><span class="s2">"[2]==out"</span> --top-column<span class="o">=</span><span class="m">1</span> --bottom-column<span class="o">=</span><span class="m">1</span> --annotate-max --value-tick-label-format<span class="o">=</span>metric --value-units<span class="o">=</span>B --title<span class="o">=</span><span class="s2">"Traffic to/from Web Servers"</span> --value-scale<span class="o">=</span>log --<span class="sb">`</span>fix-scale-min<span class="o">=</span><span class="m">1</span>
</pre></div>
<p><a href="https://www.rsreese.com/assets/log-rayon-graph.png"><img alt="log-rayon-graph" src="https://www.rsreese.com/assets/log-rayon-graph.png"></a></p>
<div class="highlight"><pre><span></span>$ cat <span class="m">2</span>-top.txt <span class="m">2</span>-btm.txt <span class="p">|</span> rytimeseries --style<span class="o">=</span>filled_lines --output-path<span class="o">=</span><span class="m">2</span>.png --top-filter<span class="o">=</span><span class="s2">"[2]==in"</span> --bottom-filter<span class="o">=</span><span class="s2">"[2]==out"</span> --top-column<span class="o">=</span><span class="m">1</span> --bottom-column<span class="o">=</span><span class="m">1</span> --annotate-max --value-tick-label-format<span class="o">=</span>metric --value-units<span class="o">=</span>B --title<span class="o">=</span><span class="s2">"Traffic to/from Web Servers"</span> --value-scale<span class="o">=</span>clog
</pre></div>
<p><a href="https://www.rsreese.com/assets/clog-rayon-graph.png"><img alt="clog-rayon-graph" src="https://www.rsreese.com/assets/clog-rayon-graph.png"></a></p>
<div class="highlight"><pre><span></span>$ cat <span class="m">2</span>-top.txt <span class="m">2</span>-btm.txt <span class="p">|</span> rytimeseries --style<span class="o">=</span>filled_lines --output-path<span class="o">=</span><span class="m">2</span>.png --top-filter<span class="o">=</span><span class="s2">"[2]==in"</span> --bottom-filter<span class="o">=</span><span class="s2">"[2]==out"</span> --top-column<span class="o">=</span><span class="m">1</span> --bottom-column<span class="o">=</span><span class="m">1</span> --annotate-max --value-tick-label-format<span class="o">=</span>metric --value-units<span class="o">=</span>B --title<span class="o">=</span><span class="s2">"Traffic to/from Web Servers"</span> --value-scale<span class="o">=</span>linear --value-min-pct<span class="o">=</span><span class="m">90</span> --value-max-pct<span class="o">=</span><span class="m">95</span>
</pre></div>
<p><a href="https://www.rsreese.com/assets/percentile-rayon-graph.png"><img alt="percentile-rayon-graph" src="https://www.rsreese.com/assets/percentile-rayon-graph.png"></a></p>
<p>There you go. A quick and dirty way to identify traffic surges to whatever services you have sitting behind your collector. Please leave any questions you have regarding this post below.</p>Online Information Security Analysis Tools and Resources2015-10-18T12:00:00-04:002015-10-18T12:00:00-04:00Stephen Reesetag:www.rsreese.com,2015-10-18:/online-information-security-analysis-tools-and-resources/A list of sites that analysts may find useful in their day-to-day analysis of indicators and threats. While verifying and searching for new sources, I came across Links and resources for malware samples, Malware Analysis and Incident Response Tools for the Frugal and Lazy, and Free Online Tools for Looking …<p>A list of sites that analysts may find useful in their day-to-day analysis of indicators and threats. While verifying and searching for new sources, I came across <a href="http://contagiodump.blogspot.com/2010/11/links-and-resources-for-malware-samples.html">Links and resources for malware samples</a>, <a href="http://postmodernsecurity.com/2015/09/11/malware-analysis-and-incident-response-tools-for-the-frugal-and-lazy/">Malware Analysis and Incident Response Tools for the Frugal and Lazy</a>, and <a href="https://zeltser.com/lookup-malicious-websites/">Free Online Tools for Looking Up Potentially Malicious Websites</a> which may also be helpful. Please let me know if you feel something is missing or broken by leaving a comment or <a href="https://www.rsreese.com/contact/">contacting me</a>.</p>
<p><strong><span class="caps">IP</span>/<span class="caps">ISP</span>/Domain, and <span class="caps">WHOIS</span> look-ups</strong></p>
<ul>
<li><a href="https://www.robtex.com">https://www.robtex.com</a> - <span class="caps">IP</span>/<span class="caps">DNS</span>/<span class="caps">WHOIS</span> look-ups</li>
<li><a href="http://centralops.net/co/">http://centralops.net/co/</a> - <span class="caps">IP</span>/<span class="caps">DNS</span>/<span class="caps">WHOIS</span> look-ups</li>
<li><a href="http://www.yougetsignal.com/tools/web-sites-on-web-server/">http://www.yougetsignal.com/tools/web-sites-on-web-server/</a> - Reverse lookup</li>
<li><a href="http://www.dshield.org/ipinfo.html?ip=8.8.8.8">http://www.dshield.org/ipinfo.html?ip=8.8.8.8</a> - Internet Storm Center DShield</li>
<li><a href="http://www.ipchecking.com">http://www.ipchecking.com</a> - <span class="caps">IP</span>/<span class="caps">DNS</span>/<span class="caps">WHO</span>-<span class="caps">IS</span> <span class="caps">GEOGRAPHIC</span> <span class="caps">IP</span> look-up</li>
<li><a href="http://www.isup.me">http://www.isup.me</a> - Check to see if site is up</li>
<li><a href="https://isc.sans.edu/port.html?port=8080">https://isc.sans.edu/port.html?port=8080</a> - Port details and usage statistics</li>
<li><a href="http://www.traceroute.org/#USA">http://www.traceroute.org/#<span class="caps">USA</span></a></li>
<li><a href="https://www.net.princeton.edu/tools">https://www.net.princeton.edu/tools</a> - Traceroute</li>
<li><a href="http://www.projecthoneypot.org/list_of_ips.php">http://www.projecthoneypot.org/list_of_ips.php</a> - IPs obtained from honeypots</li>
<li><a href="http://whois.arin.net">http://whois.arin.net</a> - <span class="caps">IP</span> Whois lookup </li>
<li><a href="http://whois.domaintools.com">http://whois.domaintools.com</a> - Reverse Whois and Whois History</li>
<li><a href="http://www.webconfs.com/domain-age.php">http://www.webconfs.com/domain-age.php</a> - Domain age</li>
<li><a href="http://www.dnsstuff.com/">http://www.dnsstuff.com</a> - <span class="caps">IP</span>/<span class="caps">DNS</span>/<span class="caps">WHO</span>-<span class="caps">IS</span> look-ups</li>
<li><a href="http://www.dnscolos.com/free-dns-report.html">http://www.dnscolos.com/free-dns-report.html</a> - <span class="caps">DNS</span> Report</li>
<li><a href="https://dnshistory.org">https://dnshistory.org</a> - The history of <span class="caps">IP</span>/<span class="caps">DNS</span> Records for domains</li>
<li><a href="http://www.dnsdigger.com">http://www.dnsdigger.com</a></li>
<li><a href="http://www.bfk.de/bfk_dnslogger_en.html">http://www.bfk.de/bfk_dnslogger_en.html</a> - Passive <span class="caps">DNS</span></li>
<li><a href="https://www.dnsdb.info">https://www.dnsdb.info</a> - <span class="caps">IP</span>/<span class="caps">DNS</span>/Passive look-ups</li>
</ul>
<p><strong><span class="caps">IP</span> and Domain analysis for malware or web-based threats</strong></p>
<ul>
<li><a href="http://www.mcafee.com/us/threat-center.aspx">http://www.mcafee.com/us/threat-center.aspx</a> - <span class="caps">IP</span> and Domain threat intel</li>
<li><a href="http://www.siteadvisor.com/sites/rsreese.com">http://www.siteadvisor.com/sites/rsreese.com</a> - McAfee Site Advisor</li>
<li><a href="https://safeweb.norton.com">https://safeweb.norton.com</a> - Norton Safe Web</li>
<li><a href="https://www.virustotal.com/#url">https://www.virustotal.com/#url</a> - Analyzes suspicious files and URLs/detects malware</li>
<li><a href="http://www.projecthoneypot.org/search_ip.php">http://www.projecthoneypot.org/search_ip.php</a> - Inspect an <span class="caps">IP</span> by Project Honey Pot</li>
<li><a href="http://urlquery.net">http://urlquery.net</a> - Detailed information about actions a browser takes while visiting a site</li>
<li><a href="http://www.dtrackr.com">http://www.dtrackr.com</a> - Domain activity tracking</li>
<li><a href="http://www.ipvoid.com">http://www.ipvoid.com</a> - Scans an <span class="caps">IP</span> address against <span class="caps">IP</span> blacklists</li>
<li><a href="http://www.urlvoid.com">http://www.urlvoid.com</a> - Scans a domain address for its reputation</li>
<li><a href="http://minotauranalysis.com">http://minotauranalysis.com</a> - Check against secure <span class="caps">DNS</span> providers and determine whether they block/redirect a hostname</li>
<li><a href="http://www.malwareurl.com/listing-urls.php">http://www.malwareurl.com/listing-urls.php</a> - Scans a domain address for its reputation</li>
<li><a href="https://sitecheck.sucuri.net">https://sitecheck.sucuri.net</a> - Check the site for malware, blacklisting status, and out-of-date software</li>
<li><a href="http://www.avgthreatlabs.com/ww-en/website-safety-reports">http://www.avgthreatlabs.com/ww-en/website-safety-reports</a>k - Check the safety of a <span class="caps">URL</span> or web page by scanning it for threats</li>
<li><a href="http://global.sitesafety.trendmicro.com">http://global.sitesafety.trendmicro.com</a> - Latest tests indicate that this website contains no malicious software and shows no signs of fraud</li>
<li><a href="http://urlblacklist.com/?sec=search">http://urlblacklist.com/?sec=search</a> - Find out if a <span class="caps">URL</span> is in the blacklist</li>
<li><a href="http://www.senderbase.org">http://www.senderbase.org</a> - Cisco <span class="caps">IP</span> and domain blacklist check </li>
</ul>
<p><strong>Open-source Threat Reports, <span class="caps">IP</span> and Domain Blacklists</strong></p>
<ul>
<li><a href="http://www.sophos.com/en-us/threat-center.aspx">http://www.sophos.com/en-us/threat-center.aspx</a> - Malware reports</li>
<li><a href="http://www.symantec.com/security_response/">http://www.symantec.com/security_response/</a> - Threats, risks, and vulnerabilities</li>
<li><a href="http://www.spamhaus.org/lookup/">http://www.spamhaus.org/lookup/</a> - Database of IPs reporting email spam abuse</li>
<li><a href="http://hosts-file.net">http://hosts-file.net</a> - Community managed host file to protect against malicious</li>
<li><a href="http://www.phishtank.com">http://www.phishtank.com</a> - PhishTank</li>
<li><a href="http://www.malwaredomainlist.com/mdl.php">http://www.malwaredomainlist.com/mdl.php</a> - Malicious domains/IPs and malware</li>
<li><a href="http://malc0de.com/database/">http://malc0de.com/database/</a> - Database of malicious domains/IPs and malware</li>
<li><a href="http://www.malwaregroup.com">http://www.malwaregroup.com</a> - Feed of malware reports from multiple sites</li>
<li><a href="http://www.mywot.com">http://www.mywot.com</a> - Tells you reputation of a website from public reports</li>
<li><a href="http://www.malwaredomains.com">http://www.malwaredomains.com</a> - Malware Prevention through Domain Blocking</li>
<li><a href="http://multirbl.valli.org">http://multirbl.valli.org</a> - Free multiple <span class="caps">DNSBL</span>/<span class="caps">RBL</span> lookup and FCrDNS check tool</li>
<li><a href="http://toolbar.netcraft.com/stats/countries">http://toolbar.netcraft.com/stats/countries</a> - Phishiest hosting countries</li>
<li><a href="http://www.dcwg.org/detect/">http://www.dcwg.org/detect/</a> - Detect <span class="caps">DNS</span> Changer infection</li>
<li><a href="http://stopmalvertising.com">http://stopmalvertising.com</a> - Investigate distribution of malware exploits through online advertising networks</li>
</ul>
<p><strong>Malware Binary Analysis</strong></p>
<ul>
<li><a href="https://www.virustotal.com/en/">https://www.virustotal.com/en/</a> - Analyze suspicious binaries</li>
<li><a href="http://anubis.iseclab.org">http://anubis.iseclab.org</a> - <span class="caps">ANUBIS</span> ANalyzing Unknown BInarieS</li>
<li><a href="http://wepawet.iseclab.org">http://wepawet.iseclab.org</a> - Analyze Flash, JavaScript, and PDFs</li>
<li><a href="http://jsunpack.jeek.org">http://jsunpack.jeek.org</a> - JavaScript Unpacker/ Decode De-Obfuscated JavaScript</li>
<li><a href="http://minotauranalysis.com">http://minotauranalysis.com</a> - Hash value search</li>
<li><a href="http://www.threatexpert.com/filescan.aspx">http://www.threatexpert.com/filescan.aspx</a> - Analyze suspicious binaries</li>
<li><a href="http://www.threattracksecurity.com/resources/sandbox-malware-analysis.aspx">http://www.threattracksecurity.com/resources/sandbox-malware-analysis.aspx</a></li>
</ul>
<p><strong>Malware Samples</strong></p>
<ul>
<li><a href="http://contagiodump.blogspot.com">http://contagiodump.blogspot.com</a></li>
<li><a href="http://contagioexchange.blogspot.com">http://contagioexchange.blogspot.com</a></li>
<li><a href="http://malware.lu">http://malware.lu</a></li>
<li><a href="http://virusshare.com">http://virusshare.com</a></li>
</ul>
<p><strong><span class="caps">HTTP</span> Agent sniffers, Decode De-Obfuscate JavaScript and Base 64</strong></p>
<ul>
<li><a href="http://web-sniffer.net">http://web-sniffer.net</a> - Analysis of <span class="caps">HTTP</span> Request and Response Headers</li>
<li><a href="http://builtwith.com">http://builtwith.com</a> - Determine services running on target</li>
<li><a href="http://www.rexswain.com/httpview.html">http://www.rexswain.com/httpview.html</a> - See <em>exactly</em> what an <span class="caps">HTTP</span> request returns to your browser</li>
<li><a href="http://gsitecrawler.com/tools/Server-Status.aspx">http://gsitecrawler.com/tools/Server-Status.aspx</a> Sever redirect checker</li>
<li><a href="http://www.unmaskcontent.com">http://www.unmaskcontent.com</a> - Unmask Content</li>
<li><a href="http://www.yellowpipe.com/yis/tools/encrypter">http://www.yellowpipe.com/yis/tools/encrypter</a> - encode/decode or encrypt/decrypt your documents in various formats such as: <span class="caps">ASCSII</span>, Binary, Base 64,<span class="caps">HTML</span>/text/JavaScript Escaping</li>
<li><a href="http://scriptasylum.com/tutorials/encode-decode.html">http://scriptasylum.com/tutorials/encode-decode.html</a> - <span class="caps">HTML</span>/text/JavaSript Escaping/Encoding Script</li>
<li><a href="http://ln.hixie.ch/?start=1073090889&count=1">http://ln.hixie.ch/?start=1073090889&count=1</a> - Unicode decoder tools</li>
<li><a href="http://www.crypo.com">http://www.crypo.com</a> - Encode or Decode strings, email and other messages</li>
<li><a href="http://spyonweb.com">http://spyonweb.com</a> - Determine what sites are sharing Google analytic code</li>
<li><a href="http://www.netdemon.net/decode.html">http://www.netdemon.net/decode.html</a> - obfuscated <span class="caps">URL</span> Decoder</li>
</ul>
<p><strong>BotNet Tracking</strong></p>
<ul>
<li><a href="http://botlab.org">http://botlab.org</a> - Spam ranking, botnet <span class="amp">&</span> C2 tracking</li>
<li><a href="https://palevotracker.abuse.ch">https://palevotracker.abuse.ch</a> - Palevo Tracker</li>
<li><a href="https://zeustracker.abuse.ch/">https://zeustracker.abuse.ch</a> - ZeuS Tracker</li>
<li><a href="https://zeustracker.abuse.ch">https://spyeyetracker.abuse.ch</a> - SpyEye Tracker</li>
<li><a href="http://atlas.arbor.net/summary/fastflux">http://atlas.arbor.net/summary/fastflux</a> - <span class="caps">ATLAS</span> Summary Report</li>
<li><a href="http://www.cert.pl/news/4711/langswitch_lang/en">http://www.cert.pl/news/4711/langswitch_lang/en</a> - ZeuS – <span class="caps">P2P</span>+<span class="caps">DGA</span> variant</li>
</ul>
<p><strong>Site History</strong> </p>
<ul>
<li><a href="https://archive.org">https://archive.org</a> - Wayback Machine Internet Archive</li>
<li><a href="http://www.spiderfoot.net">http://www.spiderfoot.net</a> - Spider Indexing</li>
</ul>
<p><strong>Google Hacking</strong></p>
<ul>
<li><a href="http://www.exploit-db.com/google-dorks/">http://www.exploit-db.com/google-dorks/</a> - Google Hacking Database (<span class="caps">GHDB</span>) by <a href="http://www.hackersforcharity.org/">HfC</a></li>
<li><a href="http://ghh.sourceforge.net">http://ghh.sourceforge.net</a> - Google Hack Honeynet</li>
<li><a href="http://www.edge-security.com">http://www.edge-security.com</a></li>
</ul>Graphing Namebench Spreadsheet Data with R2015-07-11T12:00:00-04:002015-07-11T12:00:00-04:00Stephen Reesetag:www.rsreese.com,2015-07-11:/graphing-namebench-spreadsheet-data-with-r/In the previous post, I described the process of benchmarking domain name servers for a website domain with a modified version of Namebench. Namebench generates graphs using the Google chart API. This left me wanting a little more therefore decided to explore the data using the R Project. This post …<p>In the previous <a href="/benchmarking-website-domain-name-servers/">post</a>, I described the process of benchmarking domain name servers for a website domain with a modified version of <a href="https://github.com/rsreese/namebench">Namebench</a>. Namebench generates graphs using the Google chart <span class="caps">API</span>. This left me wanting a little more therefore decided to explore the data using the <a href="http://www.r-project.org">R Project</a>. This post makes the assumption you are using our <a href="https://www.rsreese.com/assets/namebench_2015-07-14_1952.csv">data set</a> in order to follow along or else <span class="caps">YMMV</span>. </p>
<p>First, remove trailing commas from each row:</p>
<div class="highlight"><pre><span></span>$ sed <span class="s1">'s/,[[:space:]]*$//'</span> namebench_2015-07-14_1952.csv > data.csv
</pre></div>
<p>Next, we read in the data from the <span class="caps">CSV</span> file into the R buffer assuming you are already in the R console:</p>
<div class="highlight"><pre><span></span><span class="o">></span> data <span class="o"><-</span> read.table<span class="p">(</span>file<span class="o">=</span><span class="s">"data.csv"</span><span class="p">,</span>header<span class="o">=</span><span class="kc">TRUE</span><span class="p">,</span>sep<span class="o">=</span><span class="s">","</span><span class="p">,</span>row.names<span class="o">=</span><span class="kc">NULL</span><span class="p">)</span>
</pre></div>
<p>If you get errors about a line not having 9 elements, you likely had timeouts in your <span class="caps">DNS</span> queries. You can either re-run the test until you do not experience any timeouts or remove the Timeout error message lines. Something like <code>grep -v Timeout data.csv >a.out</code> and copy back to data.csv or whatever filename you would like to work with.</p>
<p>As an aside, we can also export our data back out:</p>
<div class="highlight"><pre><span></span>> write.table(data, 'a.txt', col.names=NA)
</pre></div>
<p>Which results in:</p>
<div class="highlight"><pre><span></span>"" "IP" "Name" "Test_Num" "Record" "Record_Type" "Duration" "TTL" "Answer_Count" "Response"
"1" "2600:3c01::a" "Linode 2 IPv6" 0 "www.rsreese.com." "A" 76.2228965759277 86400 1 "74.207.234.79"
"2" "2600:3c01::a" "Linode 2 IPv6" 0 "www.rsreese.com." "A" 73.7550258636475 86400 1 "74.207.234.79"
"3" "2600:3c01::a" "Linode 2 IPv6" 0 "www.rsreese.com." "A" 73.4801292419434 86400 1 "74.207.234.79"
"4" "2600:3c01::a" "Linode 2 IPv6" 0 "www.rsreese.com." "A" 76.7168998718262 86400 1 "74.207.234.79"
"5" "2600:3c01::a" "Linode 2 IPv6" 0 "www.rsreese.com." "A" 73.2970237731934 86400 1 "74.207.234.79"
"6" "2600:3c01::a" "Linode 2 IPv6" 0 "www.rsreese.com." "A" 73.3959674835205 86400 1 "74.207.234.79"
"7" "2600:3c01::a" "Linode 2 IPv6" 0 "www.rsreese.com." "A" 72.7560520172119 86400 1 "74.207.234.79"
"8" "2600:3c01::a" "Linode 2 IPv6" 0 "www.rsreese.com." "A" 76.8599510192871 86400 1 "74.207.234.79"
"9" "2600:3c01::a" "Linode 2 IPv6" 0 "www.rsreese.com." "A" 72.8960037231445 86400 1 "74.207.234.79"
"10" "2600:3c01::a" "Linode 2 IPv6" 0 "www.rsreese.com." "A" 74.0060806274414 86400 1 "74.207.234.79"
--snip--
</pre></div>
<p>Now that R has our data, we can take a quick look to ensure the columns make sense:</p>
<div class="highlight"><pre><span></span>> options(width=150)
> head(data,n=10)
> head(data,n=10)
IP Name Test_Num Record Record_Type Duration TTL Answer_Count Response
1 2600:3c01::a Linode 2 IPv6 0 www.rsreese.com. A 76.22290 86400 1 74.207.234.79
2 2600:3c01::a Linode 2 IPv6 0 www.rsreese.com. A 73.75503 86400 1 74.207.234.79
3 2600:3c01::a Linode 2 IPv6 0 www.rsreese.com. A 73.48013 86400 1 74.207.234.79
4 2600:3c01::a Linode 2 IPv6 0 www.rsreese.com. A 76.71690 86400 1 74.207.234.79
5 2600:3c01::a Linode 2 IPv6 0 www.rsreese.com. A 73.29702 86400 1 74.207.234.79
6 2600:3c01::a Linode 2 IPv6 0 www.rsreese.com. A 73.39597 86400 1 74.207.234.79
7 2600:3c01::a Linode 2 IPv6 0 www.rsreese.com. A 72.75605 86400 1 74.207.234.79
8 2600:3c01::a Linode 2 IPv6 0 www.rsreese.com. A 76.85995 86400 1 74.207.234.79
9 2600:3c01::a Linode 2 IPv6 0 www.rsreese.com. A 72.89600 86400 1 74.207.234.79
10 2600:3c01::a Linode 2 IPv6 0 www.rsreese.com. A 74.00608 86400 1 74.207.234.79
> summary(data$Duration)
Min. 1st Qu. Median Mean 3rd Qu. Max.
1.455 2.582 3.836 24.430 47.640 780.500
</pre></div>
<p>We can create an aggregated table of the data based on mean values:</p>
<div class="highlight"><pre><span></span>> aggregate(data$Duration, by=list(data$Name), FUN=mean)
Group.1 x
1 CF Erin 3.752344
2 CF Ram 6.141772
3 HE 1 2.563629
4 HE 2 2.494576
5 HE 2 IPv6 2.677688
6 HE 3 5.510935
7 HE 3 IPv6 3.057263
8 HE 4 2.982669
9 HE 4 IPv6 2.626012
10 HE 5 2.642891
11 HE 5 IPv6 2.736038
12 Linode 1 49.536158
13 Linode 1 IPv6 48.098648
14 Linode 2 75.840130
15 Linode 2 IPv6 76.885061
16 Linode 3 25.727819
17 Linode 3 IPv6 26.703984
18 Linode 4 8.020208
19 Linode 4 IPv6 7.908908
20 Linode 5 82.185041
21 Linode 5 IPv6 76.434550
</pre></div>
<p>Lets see how a boxplot looks. The graph is representative of the third command listed here, others are for reference/tinkering:</p>
<div class="highlight"><pre><span></span>> plot(data$Duration ~ data$Name, horizontal=TRUE, par(las=1))
> boxplot(data$Duration ~ data$Name, horizontal=TRUE, par(las=1), col=rainbow(10))
> boxplot(data$Duration ~ data$Name, ylim=c(0,100), horizontal=TRUE, par(las=1), col=rainbow(10))
</pre></div>
<p><a href="https://www.rsreese.com/assets/namebench-boxplot.png"><img alt="namebench-boxplot" src="https://www.rsreese.com/assets/namebench-boxplot.png"></a></p>
<p>If we zoom in a little more, the distribution of the more responsive name servers becomes apparent. I believe this graph is the best representation of the fastest name servers in the dataset:</p>
<div class="highlight"><pre><span></span>> boxplot(data$Duration ~ data$Name, ylim=c(0,10), horizontal=TRUE, par(las=1), col=rainbow(10))
</pre></div>
<p><a href="https://www.rsreese.com/assets/namebench-boxplot2.png"><img alt="namebench-boxplot2" src="https://www.rsreese.com/assets/namebench-boxplot2.png"></a></p>
<p>Alternatively, we can plot using ggplot2 if available:</p>
<div class="highlight"><pre><span></span>> library(ggplot2)
> ggplot(data=data, aes(x=Duration, y=Name, group=Name, colour=Name)) + geom_line() + geom_point()
</pre></div>
<p><a href="https://www.rsreese.com/assets/namebench-ggplot2.png"><img alt="namebench-ggplot2" src="https://www.rsreese.com/assets/namebench-ggplot2.png"></a></p>
<p>Display horizontal bar graph. I did not do a great job with the axis labels here but you get the idea:</p>
<div class="highlight"><pre><span></span><span class="o">></span> agg <span class="o"><-</span> aggregate<span class="p">(</span>data<span class="o">$</span>Duration<span class="p">,</span> by<span class="o">=</span><span class="kt">list</span><span class="p">(</span>data<span class="o">$</span>Name<span class="p">),</span> FUN<span class="o">=</span><span class="kp">mean</span><span class="p">)</span>
<span class="o">></span> sorted <span class="o"><-</span> agg<span class="p">[</span><span class="kp">with</span><span class="p">(</span>agg<span class="p">,</span> <span class="kp">order</span><span class="p">(</span>x<span class="p">)),</span> <span class="p">]</span>
<span class="o">></span> mymat <span class="o"><-</span> <span class="kp">t</span><span class="p">(</span>sorted<span class="p">[</span><span class="m">-1</span><span class="p">])</span>
<span class="o">></span> <span class="kp">colnames</span><span class="p">(</span>mymat<span class="p">)</span> <span class="o"><-</span> sorted<span class="p">[,</span> <span class="m">1</span><span class="p">]</span>
<span class="o">></span> barplot<span class="p">(</span>mymat<span class="p">,</span> horiz<span class="o">=</span><span class="kc">TRUE</span><span class="p">,</span> col<span class="o">=</span><span class="kt">c</span><span class="p">(</span><span class="s">"blue"</span><span class="p">),</span> las<span class="o">=</span><span class="m">1</span><span class="p">)</span>
</pre></div>
<p><a href="https://www.rsreese.com/assets/namebench-barplot.png"><img alt="namebench-barplot" src="https://www.rsreese.com/assets/namebench-barplot.png"></a></p>
<p>Finally, we will graph a group of values from the set and display them. We also limit the range so the graph is readable:</p>
<div class="highlight"><pre><span></span>> plot(ecdf(data$Duration[data$Name=="Linode 1"]), xlim=c(45,55), ylim=c(0,1))
</pre></div>
<p><a href="https://www.rsreese.com/assets/namebench-line.png"><img alt="namebench-line" src="https://www.rsreese.com/assets/namebench-line.png"></a></p>
<p>Please leave any questions you have regarding this post below.</p>Benchmarking Website Domain Name Servers2015-06-14T12:00:00-04:002015-06-14T12:00:00-04:00Stephen Reesetag:www.rsreese.com,2015-06-14:/benchmarking-website-domain-name-servers/This post evaluates a few methods to benchmark name servers that provide resolution of your websites domain name to its respective IP address. While DNS resolution for you domain is a small piece of the process for a user to retrieve a page, it is still important to provide the …<p>This post evaluates a few methods to benchmark name servers that provide resolution of your websites domain name to its respective <span class="caps">IP</span> address. While <span class="caps">DNS</span> resolution for you domain is a small piece of the process for a user to retrieve a page, it is still important to provide the fastest experience possible, regardless of where they are connecting from. There are several methods to benchmark: from a dedicated host, or <span class="caps">VPS</span> or a last-mile end-point such as a residential connection. Dedicated host benchmark examples would include <a href="http://www.dnsperf.com">DNSPerf</a> and <a href="https://pulse.turbobytes.com">TurboBytes Pulse</a>. While the metrics provided by these assessments may be consistent, they may not necessarily represent realistic last-mile performance the end-user would typically experience. This is because these agents are typically on backbone internet connections that have peering agreements with very low latency providers. An exception to Pulse is some of the agents are hosted at locations that would be considered last-mile but at this time the results are averaged into the mean result.</p>
<p><a href="http://www.dnsperf.com">DNSPerf:</a>
<a href="https://www.rsreese.com/assets/dnsperf.png"><img alt="dnsperf" src="https://www.rsreese.com/assets/dnsperf.png"></a></p>
<p><a href="https://pulse.turbobytes.com">TurboBytes Pulse:</a>
<a href="https://www.rsreese.com/assets/pulse.png"><img alt="pulse" src="https://www.rsreese.com/assets/pulse.png"></a></p>
<p>Last-mile metrics provide us with an idea of what our site users first request for a <span class="caps">DNS</span> lookup will be. Specifically, a request for a record that is not cached by the system performing the lookup e.g. browser, <span class="caps">OS</span>, local network <span class="caps">DNS</span> forwarder, etc. Depending on the users geographic location and network connection (latency considerations), this could be the difference of several hundred milliseconds for an initial lookup. A few ways to examine last mile <span class="caps">DNS</span> are via browser, visitor analytics, or scripts. All have their pro and cons just as backbone tests do. Examples include:</p>
<p>Chrome Browser Console:
<a href="https://www.rsreese.com/assets/chrome-console.png"><img alt="chrome-console" src="https://www.rsreese.com/assets/chrome-console.png"></a></p>
<p><a href="http://tools.pingdom.com/fpt/">Pingdom:</a>
<a href="https://www.rsreese.com/assets/pingdom.png"><img alt="pingdom" src="https://www.rsreese.com/assets/pingdom.png"></a></p>
<p><a href="http://www.webpagetest.org">WebPagetest:</a>
<a href="https://www.rsreese.com/assets/webpagetest.png"><img alt="webpagetest" src="https://www.rsreese.com/assets/webpagetest.png"></a></p>
<p><a href="http://www.google.com/analytics/">Google Analytics:</a>
<a href="https://www.rsreese.com/assets/google-analytics.png"><img alt="google-analytics" src="https://www.rsreese.com/assets/google-analytics.png"></a></p>
<p>The aforementioned tools provide some solid metrics but I wanted a way to assess my domains name servers with a large number of <span class="caps">DNS</span> requests from locations of my choosing. I modified Googles Namebench code available <a href="https://www.github.com/rsreese/namebench">here</a> to allow for on demand name server benchmarks of the specified domain(s) verse its original purpose of benchmarking name servers for general <span class="caps">DNS</span> lookups. Next I setup the zone I wanted to examine at other <span class="caps">DNS</span> providers, Hurricane Electric (<span class="caps">HE</span>) and CloudFlare (<span class="caps">CF</span>). While these two hosts are not authoritative for my domain, i.e. they are not the name servers registered with my domain registrar, they will still respond if I have setup <span class="caps">DNS</span> records. In this case I am testing www.rsreese.com from a Digital Ocean host. Again, this is not realistic as we are testing from a host on optimal network and route conditions but merely to show the output. </p>
<div class="highlight"><pre><span></span><span class="gh">Fastest individual response (in milliseconds):</span>
<span class="gh">----------------------------------------------</span>
HE 3 ## 1.45507
HE 5 ## 1.56093
HE 5 IPv6 ## 1.58811
HE 3 IPv6 ## 1.61195
HE 4 ## 1.62816
HE 2 ## 1.63388
HE 4 IPv6 ## 1.66416
HE 1 ## 1.66702
HE 2 IPv6 ## 1.70302
CF Ram ## 1.98388
CF Erin ## 1.98603
Linode 4 ### 4.05502
Linode 4 IPv6 ### 4.06694
Linode 3 ############### 19.89603
Linode 3 IPv6 ################ 21.06285
Linode 1 IPv6 ################################ 42.38200
Linode 1 ################################# 43.81895
Linode 5 IPv6 ################################################### 69.63682
Linode 2 IPv6 #################################################### 70.39404
Linode 2 ##################################################### 71.63501
Linode 5 ##################################################### 72.44182
<span class="gh">Mean response (in milliseconds):</span>
<span class="gh">--------------------------------</span>
HE 2 ## 2.49
HE 1 ## 2.56
HE 4 IPv6 ## 2.63
HE 5 ## 2.64
HE 2 IPv6 ## 2.68
HE 5 IPv6 ## 2.74
HE 4 ## 2.98
HE 3 IPv6 ## 3.06
CF Erin ## 3.75
HE 3 ### 5.51
CF Ram ### 6.14
Linode 4 IPv6 ########## 21.88
Linode 4 ########## 21.99
Linode 3 ############ 25.73
Linode 3 IPv6 ################## 40.60
Linode 1 ###################### 49.54
Linode 1 IPv6 ########################### 61.91
Linode 2 ################################# 75.84
Linode 2 IPv6 ####################################### 90.58
Linode 5 IPv6 ################################################### 117.52
Linode 5 ##################################################### 123.20
</pre></div>
<p>A similar test run from a residential internet connection providing more realistic metrics for a broadband connection:</p>
<div class="highlight"><pre><span></span><span class="gh">Fastest individual response (in milliseconds):</span>
<span class="gh">----------------------------------------------</span>
CF Erin ######## 13.67188
CF Ram ######## 13.88907
HE 1 ########### 18.39805
HE 5 ########### 18.72587
HE 2 ########### 18.76688
HE 4 ########### 18.77284
HE 3 ########### 18.92185
Linode 4 ############# 21.31605
Linode 3 ################# 28.21493
Linode 1 ############################ 48.61617
Linode 2 ################################################# 83.48799
Linode 5 ##################################################### 92.02409
<span class="gh">Mean response (in milliseconds):</span>
<span class="gh">--------------------------------</span>
HE 3 ############## 27.24
HE 4 ############## 27.68
HE 1 ############### 28.58
HE 5 ############### 28.58
HE 2 ############### 28.89
Linode 4 ################ 31.83
CF Erin ################### 38.02
Linode 3 ########################## 51.06
CF Ram ########################## 51.32
Linode 1 #################################### 71.66
Linode 5 ################################################### 102.22
Linode 2 ##################################################### 106.29
</pre></div>
<p>Based on the results that I tested from a number of locations, Hurricane Electric and Cloudflare were consistently faster than Linode, whom hosts my authoritative name servers, i.e. the servers that will respond if no one upstream has the answer cached. Lastly, the Namebench tool does have some built in graphing capability as shown below (not representative of the tabular data above):</p>
<p><a href="https://www.rsreese.com/assets/namebench-chart.png"><img alt="namebench-chart" src="https://www.rsreese.com/assets/namebench-chart.png"></a></p>
<p>If you find yourself wanting a little more in the way of graphs then what the Google chart <span class="caps">API</span> provides, Namebench provides us a handy spreadsheet that we can use for graphing in R which we demonstrate in the next blog <a href="/graphing-namebench-spreadsheet-data-with-r/">post</a>. Until then, please leave any questions you have regarding this post below or file an issue on the Github page relevant to the issue you are having with the customized <a href="https://www.github.com/rsreese/namebench">Namebench</a> tool.</p>Building Apache and ModSecurity from source2015-02-27T08:00:00-05:002015-02-27T08:00:00-05:00Stephen Reesetag:www.rsreese.com,2015-02-27:/building-apache-and-modsecurity-from-source/This entry describes settting up ModSecurity on a node in order to protect a few WordPress sites I host. There are a slew of guides out there describing ModSecurity builds but I wanted to leverage the latest ModSecurity and Apache MPM Event packages which typically are not included in most …<p>This entry describes settting up <a href="http://modsecurity.org">ModSecurity</a> on a node in order to protect a few WordPress sites I host. There are a slew of guides out there describing ModSecurity <a href="https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#Installation_for_Apache">builds</a> but I wanted to leverage the latest ModSecurity and Apache <span class="caps">MPM</span> Event packages which typically are not included in most Linux distribution repositories. We use a proxy node that passes requests to the backend (origin) server hosting the web application. You may just as easily build ModSecurity on the same host that is serving your content verse using a reverse proxy, i.e. there are a number of ways to architect the setup. In the figure below, a request is first received by the proxy with ModSecurity enabled, and then passed to the origin host serving the actual content if ModSecurity does not intervene. We use Debian but other distributions should be similar.</p>
<p><img alt="ModSecurity Proxy" src="https://www.rsreese.com/assets/modsecurity-proxy-figure.png"></p>
<p>Install prerequisite packages:</p>
<div class="highlight"><pre><span></span>$ sudo apt-get install gcc libpcre3-dev libxml2-dev libcurl4-gnutls-dev
</pre></div>
<p>Download, Build, and Install <span class="caps">SSL</span> (enable shared if on 64bit):</p>
<div class="highlight"><pre><span></span>$ mkdir install
$ ./config shared --prefix<span class="o">=</span>/root/openssl-1.0.2a/install/
$ make depend
$ make
$ make <span class="nb">test</span>
$ make install
</pre></div>
<p>Download latest Apache, <span class="caps">APR</span>, and <span class="caps">APR</span> Util packages. Extract <span class="caps">APR</span> and <span class="caps">APR</span> Util, copy both to Apache src directory, build and install Apache:</p>
<div class="highlight"><pre><span></span>$ cp -R apr-util-1.5.4 httpd-2.4.12/srclib/apr-util/
$ cp -R apr-1.5.1 httpd-2.4.12/srclib/apr/
$ ./configure --with-included-apr --enable-ssl --enable-ssl-staticlib-deps --with-ssl<span class="o">=</span>/root/openssl-1.0.2/install/ --enable-proxy --with-mpm<span class="o">=</span>event
$ make
$ sudo make install
</pre></div>
<p>Download, Build and install ModSecurity (optionally install <span class="caps">LUA</span> if desired):</p>
<div class="highlight"><pre><span></span>$ tar xzf modsecurity-2.9.0.tar.gz
$ <span class="nb">cd</span> modsecurity-2.9.0/
$ ./configure --with-apxs<span class="o">=</span>/usr/local/apache2/bin/apxs --with-apr<span class="o">=</span>/root/httpd-2.4.12/srclib/apr/ --with-apu<span class="o">=</span>/root/httpd-2.4.12/srclib/apr-util/ --with-lua<span class="o">=</span>/usr/lib/x86_64-linux-gnu/pkgconfig/
$ make
$ sudo make install
</pre></div>
<p>Grab a rule-set. You may also choose to use <span class="caps">GIT</span> to download.</p>
<div class="highlight"><pre><span></span>$ wget https://github.com/SpiderLabs/owasp-modsecurity-crs/tarball/master
$ mv master master.tar.gz
$ cp -R SpiderLabs-owasp-modsecurity-crs-ebe8790/ /usr/local/apache2/conf/crs/
$ <span class="nb">cd</span> /usr/local/apache2/conf/crs/
$ mv modsecurity_crs_10_setup.conf.example modsecurity_crs_10_setup.conf
$ ln -s /usr/local/apache2/conf/crs/modsecurity_crs_10_setup.conf activated_rules/
$ <span class="k">for</span> f in <span class="sb">`</span>ls base_rules/<span class="sb">`</span> <span class="p">;</span> <span class="k">do</span> ln -s /usr/local/apache2/conf/crs/base_rules/<span class="nv">$f</span> activated_rules/<span class="nv">$f</span> <span class="p">;</span> <span class="k">done</span>
$ <span class="k">for</span> f in <span class="sb">`</span>ls optional_rules/<span class="sb">`</span> <span class="p">;</span> <span class="k">do</span> ln -s /usr/local/apache2/conf/crs/optional_rules/<span class="nv">$f</span> activated_rules/<span class="nv">$f</span> <span class="p">;</span> <span class="k">done</span>
$ mkdir /etc/modsec
$ <span class="nb">cd</span>
$ cp modsecurity-2.9.0/modsecurity.conf-recommended /etc/modsec/modsecurity.conf
$ cp modsecurity-2.9.0/unicode.mapping /etc/modsec/
$ vim /etc/modsec/whitelist.conf
</pre></div>
<p>Setup your Apache site, virtual host, or use proxy pass in order to fetch from a back-end origin node. Add ModSecurity directives to Apache conf file:</p>
<div class="highlight"><pre><span></span>LoadModule unique_id_module modules/mod_unique_id.so
LoadModule security2_module modules/mod_security2.so
<span class="nt"><IfModule</span> <span class="err">security2_module</span><span class="nt">></span>
Include /etc/modsec/modsecurity.conf
Include conf/crs/activated_rules/*.conf
Include /etc/modsec/whitelist.conf
SecRule ARGS "mod_security_test" "t:normalisePathWin,id:99999,severity:4,msg:'Drive Access'"
<span class="nt"></IfModule></span>
</pre></div>
<p>Start Apache and test to validate rules are logging and optionally being enforced. You should see a 403 forbidden response meaning that the malicious requests were blocked. Now you can move to tuning the ruleset to your web application:</p>
<p>http://waf.rsreese.com/?test=mod_security_test</p>
<p><img alt="Forbidden" src="https://www.rsreese.com/assets/modsecurity-forbidden.png"></p>
<p>If something is not clear, leave a comment below.</p>Redirect HTTP to HTTPS using Varnish2014-12-30T08:00:00-05:002014-12-30T08:00:00-05:00Stephen Reesetag:www.rsreese.com,2014-12-30:/redirect-http-to-https-using-varnish/I recently enabled HTTPS on this site and wanted to use a 301 redirect in order to correctly re-route guests from HTTP to HTTPS (HTTP to SSL/TLS). I originally performed all of my rewrites in Apache which acts as my backend. While Apache handled the typical non-www to www …<p>I recently enabled <span class="caps">HTTPS</span> on this site and wanted to use a 301 redirect in order to correctly re-route guests from <span class="caps">HTTP</span> to <span class="caps">HTTPS</span> (<span class="caps">HTTP</span> to <span class="caps">SSL</span>/<span class="caps">TLS</span>). I originally performed all of my rewrites in Apache which acts as my backend. While Apache handled the typical non-www to www redirects with ease, it created a redirect loop when attempting to redirect users from <span class="caps">HTTP</span> to <span class="caps">HTTPS</span>. I decided to let Varnish Cache 4 rather than the Apache backend handle the redirect.</p>
<p><img alt="HTTP to HTTPS redirect" src="https://www.rsreese.com/assets/http-to-https.png"></p>
<p>The documentation on the on the Varnish site is for Varnish 3 which is not compatible for Varnish 4 as of this writing:</p>
<div class="highlight"><pre><span></span><span class="nt">sub</span> <span class="nt">vcl_recv</span> <span class="p">{</span>
<span class="err">if</span> <span class="err">(</span> <span class="err">(req.http.host</span> <span class="err">~</span> <span class="err">"^(?i)somesite.org"</span> <span class="err">||</span> <span class="err">req.http.host</span> <span class="err">~</span> <span class="err">"^(?i)www.somesite.org")</span>
<span class="err">&&</span> <span class="err">req.http.X-Forwarded-Proto</span> <span class="err">!~</span> <span class="err">"(?i)https")</span> <span class="err">{</span>
<span class="err">set</span> <span class="err">req.http.x-Redir-Url</span> <span class="err">=</span> <span class="err">"</span><span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">www</span><span class="o">.</span><span class="n">somesite</span><span class="o">.</span><span class="n">org</span><span class="err">"</span> <span class="o">+</span> <span class="n">req</span><span class="o">.</span><span class="n">url</span><span class="p">;</span>
<span class="err">error</span> <span class="err">750</span> <span class="err">req.http.x-Redir-Url</span><span class="p">;</span>
<span class="p">}</span>
<span class="err">}</span>
<span class="nt">sub</span> <span class="nt">vcl_error</span> <span class="p">{</span>
<span class="err">if</span> <span class="err">(obj.status</span> <span class="err">==</span> <span class="err">750)</span> <span class="err">{</span>
<span class="err">set</span> <span class="err">obj.http.Location</span> <span class="err">=</span> <span class="err">obj.response</span><span class="p">;</span>
<span class="err">set</span> <span class="err">obj.status</span> <span class="err">=</span> <span class="err">302</span><span class="p">;</span>
<span class="err">return</span> <span class="err">(deliver)</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>After some research, I found a <a href="http://www.softprayog.in/troubleshooting/how-to-redirect-non-www-urls-to-www-in-varnish">redirect example</a> that was similar to what I was trying to achieve in Varnish 4:</p>
<div class="highlight"><pre><span></span><span class="nt">sub</span> <span class="nt">vcl_recv</span> <span class="p">{</span>
<span class="err">if</span> <span class="err">(</span> <span class="err">(req.http.host</span> <span class="err">~</span> <span class="err">"^(?i)www.domain.com"</span> <span class="err">||</span> <span class="err">req.http.host</span> <span class="err">~</span> <span class="err">"^(?i)domain.com")</span> <span class="err">&&</span> <span class="err">req.http.X-Forwarded-Proto</span> <span class="err">!~</span> <span class="err">"(?i)https")</span> <span class="err">{</span>
<span class="err">return</span> <span class="err">(synth(750,</span> <span class="err">""))</span><span class="p">;</span>
<span class="p">}</span>
<span class="err">}</span>
<span class="nt">sub</span> <span class="nt">vcl_synth</span> <span class="p">{</span>
<span class="err">if</span> <span class="err">(resp.status</span> <span class="err">==</span> <span class="err">750)</span> <span class="err">{</span>
<span class="err">set</span> <span class="err">resp.status</span> <span class="err">=</span> <span class="err">301</span><span class="p">;</span>
<span class="err">set</span> <span class="err">resp.http.Location</span> <span class="err">=</span> <span class="err">"</span><span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">www</span><span class="o">.</span><span class="n">domain</span><span class="o">.</span><span class="n">com</span><span class="err">"</span> <span class="o">+</span> <span class="n">req</span><span class="o">.</span><span class="n">url</span><span class="p">;</span>
<span class="err">return(deliver)</span><span class="p">;</span>
<span class="p">}</span>
<span class="err">}</span>
</pre></div>
<p>Now non-<span class="caps">HTTPS</span> requests to domains listed in the <code>vcl_recv</code> should redirect to the respective <span class="caps">HTTPS</span> version of your site.</p>Making WordPress Fast2014-12-15T12:00:00-05:002014-12-15T12:00:00-05:00Stephen Reesetag:www.rsreese.com,2014-12-15:/making-wordpress-fast/This site previously used WordPress as a CMS platform. Quite a bit of time was spent tuning in order to get page load times that were consistently less then 500ms although usually closer to 200 to 300ms. The WordPress site was able to burst to around to 2000 users per …<p>This site previously used WordPress as a <span class="caps">CMS</span> platform. Quite a bit of time was spent tuning in order to get page load times that were consistently less then 500ms although usually closer to 200 to 300ms. The WordPress site was able to burst to around to 2000 users per <a href="https://www.blitz.io/report/904c1ab082af0d106e4be71c0023b2aa">blitz.io</a> although there was an error here or there. I speculate that is a limitation on the outgoing bandwidth of the <span class="caps">VPS</span> as the system resources appeared stable. The largest improvements were made by caching, Varnish prevented repeat Apache, <span class="caps">PHP</span> and MySQL requests. Real world saturation is a different scenario but not too bad when compared to a stock <a href="http://en.wikipedia.org/wiki/LAMP_%28software_bundle%29"><span class="caps">LAMP</span></a> setup. A stock <span class="caps">LAMP</span> setup on my <span class="caps">VPS</span> would handle around 50 req/sec before services started to hang up. I have since migrated to a new platform but wanted to keep this around for reference.</p>
<p>Note that the process flow in the following diagram is approximate and does not necessarily accurate.</p>
<p><img alt="Web Stack Architecture" src="https://www.rsreese.com/assets/web-stack-arch.png"></p>
<ul>
<li>Hosted with <a href="http://www.linode.com/?r=6579d0b21f581ea769a6ca4af46de0dad6f88df8">Linode</a> on a <a href="http://wiki.xenproject.org/wiki/Xen_Overview">Xen Hypervisor</a>.</li>
<li><a href="http://www.debian.org/">Debian</a> Linux Operating System</li>
<li><a href="http://httpd.apache.org/docs/2.2/mod/worker.html">Apache <span class="caps">MPM</span> worker</a> - Multi-Processing Module implementing a hybrid multi-threaded multi-process web server</li>
<li><a href="https://developers.google.com/speed/pagespeed/mod">Google PageSpeed Apache module</a> backed by Memcached</li>
<li><a href="http://php-fpm.org/"><span class="caps">PHP</span>-<span class="caps">FPM</span></a> (FastCGI Process Manager)</li>
<li><a href="https://www.varnish-cache.org/">Varnish Cache</a> - a web application accelerator <span class="caps">AKA</span> a caching <span class="caps">HTTP</span> reverse proxy</li>
<li><a href="http://memcached.org/">Memcached</a> memory object caching system</li>
<li><a href="http://wordpress.org/">WordPress</a> running 2010 template</li>
<li><a href="http://wordpress.org/extend/plugins/batcache/">Batcache</a> plugin using <a href="http://svn.wp-plugins.org/memcached/trunk/">Object Cache</a> backed by Memcached</li>
</ul>
<p>If you are looking for a solid do it yourself hosting provider, I recommend you checkout <a href="http://www.linode.com/?r=6579d0b21f581ea769a6ca4af46de0dad6f88df8">Linode</a> and use my referral <a href="http://www.linode.com/?r=6579d0b21f581ea769a6ca4af46de0dad6f88df8">link</a> if you sign up which helps support this site.</p>
<p><img alt="Blitz Report" src="https://www.rsreese.com/assets/wordpress-blitz-report.png"></p>Parsing Microsoft DNS Server Logs2014-12-14T12:00:00-05:002014-12-14T12:00:00-05:00Stephen Reesetag:www.rsreese.com,2014-12-14:/parsing-microsoft-dns-server-logs/This is a quick post about one of many ways you may want to parse Microsoft DNS server logs. I this case, I simply wanted to know the top talkers. We use shell and Python in this entry on a Linux host. We follow-up with an all inclusive Python script …<p>This is a quick post about one of many ways you may want to parse Microsoft <span class="caps">DNS</span> server logs. I this case, I simply wanted to know the top talkers. We use shell and Python in this entry on a Linux host. We follow-up with an all inclusive Python script if you want to skip to the end.</p>
<p>Here is the example data or you can follow along with your own:</p>
<div class="highlight"><pre><span></span>DNS Server log file creation at 6/15/2014 6:11:48 PM UTC
Log file wrap at 6/15/2014 5:00:23 PM
Message logging key (for packets - other items use a subset of these fields):
Field # Information Values
------- ----------- ------
1 Date^M
2 Time^M
3 Thread ID
4 Context
5 Internal packet identifier^M
6 UDP/TCP indicator^M
7 Send/Receive indicator^M
8 Remote IP^M
9 Xid (hex)^M
10 Query/Response R = Response^M
blank = Query^M
11 Opcode Q = Standard Query^M
N = Notify^M
U = Update^M
? = Unknown^M
12 [ Flags (hex)^M
13 Flags (char codes) A = Authoritative Answer^M
T = Truncated Response^M
D = Recursion Desired^M
R = Recursion Available^M
14 ResponseCode ]^M
15 Question Type^M
16 Question Name^M
20140816 16:08:57 588 PACKET 019B99F0 UDP Rcv 192.168.0.2 80fd Q [0001 D NOERROR] A (3)www(1)l(6)google(3)com(0)
20140816 16:08:57 588 PACKET 019CEFF0 UDP Snd 192.168.0.2 622d Q [0001 D NOERROR] A (3)www(1)l(6)google(3)com(0)
20140816 16:08:57 588 PACKET 01C61480 UDP Rcv 192.168.0.2 622d R Q [8081 DR NOERROR] A (3)www(1)l(6)google(3)com(0)
20140816 16:08:57 588 PACKET 01C61480 UDP Snd 192.168.0.2 80fd R Q [8081 DR NOERROR] A (3)www(1)l(6)google(3)com(0)
20140816 15:51:47 588 PACKET 02131B00 UDP Snd 192.168.0.2 1b77 Q [0001 D NOERROR] A (9)messaging(9)microsoft(3)com(0)
20140816 15:51:47 588 PACKET 0242BD70 UDP Rcv 192.168.0.2 1b77 R Q [8081 DR NOERROR] A (9)messaging(9)microsoft(3)com(0)
20140816 16:28:56 588 PACKET 02447E50 UDP Rcv 192.168.0.2 6a24 Q [0001 D NOERROR] A (10)akamaiedge(3)net(0)
20140816 16:28:56 588 PACKET 01E8B070 UDP Snd 192.168.0.2 f11d Q [0001 D NOERROR] A (10)akamaiedge(3)net(0)
20140816 16:28:56 588 PACKET 01BDA5A0 UDP Rcv 192.168.0.2 f11d R Q [8081 DR NOERROR] A (10)akamaiedge(3)net(0)
20140816 16:28:56 588 PACKET 01BDA5A0 UDP Snd 192.168.0.2 6a24 R Q [8081 DR NOERROR] A (10)akamaiedge(3)net(0)
</pre></div>
<p>Since there is a header, cut the 28 header lines.</p>
<div class="highlight"><pre><span></span>$ sed <span class="s1">'1,29d'</span> log
</pre></div>
<p>Convert log from Windows to Unix format to handle pesky line returns:</p>
<div class="highlight"><pre><span></span>$ awk <span class="s1">'{ sub("\r$", ""); print }'</span> log > log.wintounix
</pre></div>
<p>Get rid of blank lines:</p>
<div class="highlight"><pre><span></span>$ sed <span class="s1">'/^$/d'</span> log.wintounix > log.nolines
</pre></div>
<p>Python code we are going to use to parse the file we have cleaned up. </p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">re</span>
<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">Counter</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'log.nolines'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">Counter</span><span class="p">(</span><span class="s1">'.'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="sa">r</span><span class="s1">'(\w+\(\d+\))'</span><span class="p">,</span><span class="n">line</span><span class="o">.</span><span class="n">split</span><span class="p">()[</span><span class="o">-</span><span class="mi">1</span><span class="p">])[</span><span class="o">-</span><span class="mi">2</span><span class="p">:])</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">f</span><span class="p">)</span>
<span class="k">for</span> <span class="n">domain</span><span class="p">,</span> <span class="n">count</span> <span class="ow">in</span> <span class="n">c</span><span class="o">.</span><span class="n">most_common</span><span class="p">():</span>
<span class="k">print</span> <span class="n">domain</span><span class="p">,</span><span class="n">count</span>
</pre></div>
<p>Sort the values returned from the Python script above, modify the key as needed.</p>
<div class="highlight"><pre><span></span>$ sort -t<span class="s2">" "</span> -k3 -n -r parsed > parsed.sorted
</pre></div>
<p>That was a lot of work to parse a file. Lets make it a little easier. Run the following with an input file: <code>parseMSDNS.py log</code></p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env python</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">fileinput</span>
<span class="kn">import</span> <span class="nn">operator</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="n">ret</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">filename</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">myfile</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span><span class="s1">'r'</span><span class="p">)</span>
<span class="n">start_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="k">with</span> <span class="n">myfile</span> <span class="k">as</span> <span class="n">theFile</span><span class="p">:</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">theFile</span><span class="p">:</span>
<span class="c1"># normalize newlines</span>
<span class="c1">#line = line.replace('\r\n', '\n').line.replace('\r', '\n')</span>
<span class="c1"># match pattern returns true of false</span>
<span class="n">match</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="sa">r</span><span class="s1">'Q \[.+\].+\(\d+\)([^\(]+)\(\d+\)([^\(]+)'</span><span class="p">,</span><span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span>
<span class="k">if</span> <span class="n">match</span> <span class="o">!=</span> <span class="bp">None</span><span class="p">:</span>
<span class="c1"># if a match, determine the value</span>
<span class="n">key</span> <span class="o">=</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">match</span><span class="o">.</span><span class="n">groups</span><span class="p">())</span>
<span class="c1"># calculate the number of key</span>
<span class="k">if</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">ret</span><span class="o">.</span><span class="n">keys</span><span class="p">():</span>
<span class="n">ret</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">ret</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">ret</span><span class="o">.</span><span class="n">keys</span><span class="p">(),</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">k</span><span class="p">:</span><span class="n">ret</span><span class="p">[</span><span class="n">k</span><span class="p">],</span> <span class="n">reverse</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
<span class="k">print</span> <span class="s2">"{:15} - {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">ret</span><span class="p">[</span><span class="n">k</span><span class="p">])</span>
<span class="k">print</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">start_time</span><span class="p">,</span> <span class="s2">"seconds"</span>
</pre></div>
<p>That should do it. Leave a comment if something is not working as expected.</p>Parsing Netflow using Kibana via Logstash to ElasticSearch2014-03-18T02:40:00-04:002014-03-18T02:40:00-04:00Stephen Reesetag:www.rsreese.com,2014-03-18:/parsing-netflow-using-kibana-via-logstash-to-elasticsearch/This blog entry shows how to easily insert flow data into an ElasticSearch instance using Logstash and view the data using Kibana. To keep the example simple, we will use Kibana that is integrated in LogStash. We will not use the ElasticSearch that is bundled with LogStash. Instead, we will …<p>This blog entry shows how to easily insert flow data into an
<a href="http://www.elasticsearch.org/">ElasticSearch</a> instance using <a href="http://logstash.net/">Logstash</a> and view the data using
<a href="http://www.elasticsearch.org/overview/kibana/">Kibana</a>. To keep the example simple, we will use Kibana that is
integrated in LogStash. We will not use the ElasticSearch that is
bundled with LogStash. Instead, we will run latest stable version of
ElasticSearch. Testing for this entry was done using Ubuntu 12.04 but
most Linux or similar distributions should work fine.</p>
<p>First, I needed the ability to generate network flow. Softflowd provided
a simple solution for my purposes. You skip the flow generation
installation if you already have a v5 or v9 netflow source you could
point to your LogStash instance. My testing was done with netflow
version 9, but it appears the the LogStash netflow codec will also
support 5. Softflowd required, byacc which you can get from <a href="http://invisible-island.net/byacc/byacc.html#download">here</a>.</p>
<div class="highlight"><pre><span></span>$ ./configure
$ make
$ sudo make install
</pre></div>
<p>Next, setup the netflow daemon that will create flow records from
traffic on an interface that is designated. You can download the
Softflowd source from <a href="https://code.google.com/p/softflowd/">here</a>.</p>
<div class="highlight"><pre><span></span>$ ./configure
$ make
$ sudo ./softflowd -i eth0 -n <span class="m">127</span>.0.0.1:12345 -v <span class="m">9</span> -d
</pre></div>
<p>Before running ElasticSearch or LogStash, you will need Java. The latest
7.0 Java version should work just fine. You can confirm your Java version:</p>
<div class="highlight"><pre><span></span>$ java -version
</pre></div>
<p>Before we run LogStash, grab the latest <a href="http://www.elasticsearch.org/downloads/">ElasticSearch</a> version from
the 0.90.x train. While ElasticSearch 1.x is out, I do not believe
LogStash is yet compatible. If need be, you can edit the memory
requirements in the following configuration file:</p>
<div class="highlight"><pre><span></span>$ vim ./elasticsearch-0.90.12/bin/elasticsearch.in.sh
</pre></div>
<p>Next start the ElasticSearch instance:</p>
<div class="highlight"><pre><span></span>$ sudo ./elasticsearch-0.90.12/bin/elasticsearch
</pre></div>
<p>Pull the latest <a href="http://logstash.net/">LogStash</a> <span class="caps">JAR</span>, before trying to run it, you
will need a netflow configuration file. This configuration file says
that we expect to receive network flow on <span class="caps">UDP</span> port 12345. Secondly, we
output to <span class="caps">STDOUT</span> and the ElasticSearch entry, the former output is for testing.</p>
<div class="highlight"><pre><span></span>input {
udp {
port => 12345
codec => netflow
}
}
output {
stdout { }
elasticsearch { host => "127.0.0.1" }
}
</pre></div>
<p>Next, we begin collecting netflow:</p>
<div class="highlight"><pre><span></span>$ sudo java -jar ./Downloads/logstash-1.3.3-flatjar.jar agent -f logstash/netflow.conf -- <span class="p">&</span>
</pre></div>
<p>After a minute or two, you should start seeing some entries via <span class="caps">STDOUT</span>
in the terminal you started LogStash in. While you could start Kibana
with the previous entry by adding the <em>web</em> toggle, I preferred separate
instances for my evaluation:</p>
<div class="highlight"><pre><span></span>$ sudo java -jar ./Downloads/logstash-1.3.3-flatjar.jar agent web -- <span class="p">&</span>
</pre></div>
<p>Lastly, the fun part, you should be able to cruise over to either
localhost or whatever <span class="caps">IP</span> address the systems as appending by port 9292
and starting tinkering:</p>
<div class="highlight"><pre><span></span>http://127.0.0.1:9292
</pre></div>
<p>Here are three dashboards I quickly put together. Not only is Logstash a
good way to quickly parse netflow, the dashboard shiny:</p>
<p><a href="https://www.rsreese.com/assets/kibana1.png"><img alt="Kibana Screen Shot" src="https://www.rsreese.com/assets/kibana1-thumb.png"></a></p>
<p><a href="https://www.rsreese.com/assets/kibana2.png"><img alt="Kibana Screen Shot" src="https://www.rsreese.com/assets/kibana2-thumb.png"></a></p>
<p><a href="https://www.rsreese.com/assets/kibana3.png"><img alt="Kibana Screen Shot" src="https://www.rsreese.com/assets/kibana3-thumb.png"></a></p>
<p>Leave a comment below if you have any questions.</p>Detecting Tor network traffic with YaF and Python2014-02-19T03:36:00-05:002014-02-19T03:36:00-05:00Stephen Reesetag:www.rsreese.com,2014-02-19:/detecting-tor-network-traffic-with-yaf-and-python/This entry continues a series of posts on identifying Tor network traffic and usage. The entry will demonstrate how to parse the output of YaF records via mediator using a Python script in order to determine if the SSL certificate values match the pattern of Tor certificates. It is assumed …<p>This entry continues a series of <a href="http://www.rsreese.com/tag/tor/">posts</a> on identifying Tor network
traffic and usage. The entry will demonstrate how to parse the output of
YaF records via mediator using a Python script in order to determine if
the <span class="caps">SSL</span> certificate values match the pattern of Tor certificates. It is
assumed you have downloaded, compiled and installed <a href="http://tools.netsa.cert.org/yaf/">YaF</a>,
<a href="https://tools.netsa.cert.org/confluence/display/tt/YAF+2.x+IPFIX+File+Mediator">mediator</a>, and <a href="https://tools.netsa.cert.org/fixbuf/">libfixbuf</a>. Please see prior <a href="http://www.rsreese.com/tag/silk/">posts</a> on this
topic or the respective documentation for installation help if needed.</p>
<p>We first generate the YaF records from the <span class="caps">PCAP</span> we acquired. You can
grab the example <span class="caps">PCAP</span> from <a href="http://www.cloudshark.org/captures/96ed6d98c159">cloudshark</a>.</p>
<div class="highlight"><pre><span></span>$ yaf --in tor.pcap --out tor.yaf
</pre></div>
<p>Next, parse the YaF output using mediator to disk in a format that we
can parse. Alternatively, we could output to MySQL verse flat text files.</p>
<div class="highlight"><pre><span></span>$ yaf_file_mediator-1.1.0/yaf_file_mediator --input tor.yaf --output tor.txt
**** Total flow count is <span class="m">29</span> ****
**** Stats Total Count is <span class="m">1</span> ****
</pre></div>
<p>Using Python, we can parse the records for patterns that match Tor <span class="caps">SSL</span> certificates.</p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/python</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="n">filename</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">myfile</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span><span class="s1">'r'</span><span class="p">)</span>
<span class="n">sourceIP</span> <span class="o">=</span> <span class="s1">'Source IP:'</span>
<span class="n">destIP</span> <span class="o">=</span> <span class="s1">'Destination IP:'</span>
<span class="n">issuerID</span> <span class="o">=</span> <span class="s1">'Issuer ID:'</span>
<span class="n">subjectID</span> <span class="o">=</span> <span class="s1">'Subject ID:'</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">myfile</span><span class="o">.</span><span class="n">readlines</span><span class="p">():</span>
<span class="n">line</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="k">if</span> <span class="n">line</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">sourceIP</span><span class="p">):</span>
<span class="n">sourceIPline</span> <span class="o">=</span> <span class="n">line</span>
<span class="k">elif</span> <span class="n">line</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">destIP</span><span class="p">):</span>
<span class="n">destIPline</span> <span class="o">=</span> <span class="n">line</span>
<span class="k">elif</span> <span class="n">line</span> <span class="ow">and</span> <span class="n">line</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">issuerID</span><span class="p">):</span>
<span class="n">issuerDomain</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="sa">r</span><span class="s1">'www.\w+.com'</span><span class="p">,</span> <span class="n">line</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">line</span> <span class="ow">and</span> <span class="n">line</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">subjectID</span><span class="p">):</span>
<span class="n">subjectDomain</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="sa">r</span><span class="s1">'www.\w+.net'</span><span class="p">,</span> <span class="n">line</span><span class="p">)</span>
<span class="k">if</span> <span class="n">issuerDomain</span> <span class="ow">and</span> <span class="n">subjectDomain</span><span class="p">:</span>
<span class="k">print</span> <span class="p">(</span><span class="n">sourceIPline</span><span class="p">)</span>
<span class="k">print</span> <span class="p">(</span><span class="n">destIPline</span><span class="p">)</span>
<span class="k">print</span> <span class="n">issuerDomain</span><span class="o">.</span><span class="n">group</span><span class="p">()</span>
<span class="k">print</span> <span class="n">subjectDomain</span><span class="o">.</span><span class="n">group</span><span class="p">()</span>
<span class="k">print</span>
<span class="n">myfile</span><span class="o">.</span><span class="n">close</span>
</pre></div>
<p>The following is an example output from the example <span class="caps">PCAP</span> provided earlier in this post. The Python regular expression ignores other <span class="caps">SSL</span> certificate values as they traditionally do not match the pattern that Tor certificates use, the inclusion of a domain for the Issuer and Subject IDs. That said, false-positives could be introduced.</p>
<div class="highlight"><pre><span></span>$ tor-ssl-parser.py tor.txt
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">198</span>.27.97.223
www.axslhtfqq.com
www.hkkch64skp7am.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">96</span>.127.153.58
www.rtqtkopfct767ai.com
www.facp2b2y5wjffbo5ioy.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">192</span>.151.147.5
www.5m6ywj2w7zs.com
www.iolbr3jbfs.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">66</span>.18.12.197
www.igdpzct5tauwgyqs.com
www.4tdznzbrfuv.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">64</span>.62.249.222
www.3pzqe4en5.com
www.glk3fwiz6.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">212</span>.83.158.173
www.lvv4l6sx3qafei2s5u.com
www.vznlngjz7a2fpg.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">212</span>.83.155.250
www.mbrdx4tz2ob5wlvazlr.com
www.shxl35n3zt.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">212</span>.83.140.45
www.3pxivyds.com
www.nolspqtib3ix.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">212</span>.83.158.50
www.s426lumoi7.com
www.ouzbot23a6lw3vvmszx.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">212</span>.83.158.40
www.3eexfeaw.com
www.iedhzej4tie4egm.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">212</span>.83.158.5
www.2fwld67ac2.com
www.6suxdq3miwwewq4.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">31</span>.7.186.228
www.5orbut4ufhohm5rlj47.com
www.orutxjqwf.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">216</span>.66.85.146
www.6pp7bfbdywvcaicqmfq.com
www.g6oa3qdobmdgl5tprm.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">178</span>.254.35.132
www.hbwpqbx4zimtptui.com
www.77wneeix55t.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">188</span>.40.98.96
www.ozsx22b4nda.com
www.lr7s5k3n6ber.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">80</span>.100.45.156
www.npmxal2ohuefme26yf.com
www.c7kriuquvh.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">91</span>.143.91.174
www.zcgg5yiwzajal4.com
www.55a4kx5jrqxezvk.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">85</span>.17.122.80
www.plgx26wgyroot37x3ysj.com
www.xwx5gpj5t2msq3.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">88</span>.159.20.120
www.s5rc22gpzrwt4e.com
www.qzsg2ioaoplbs2gaha5.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">37</span>.59.150.178
www.vywbff5wkza6npkd5l.com
www.ugdrrog5ro5wdfddj.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">91</span>.219.237.229
www.twngp3xrqgo4p.com
www.znskvp5k5pns22y2.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">95</span>.211.225.167
www.75ba5lymxpbhw3a2kb.com
www.rnspic4yus5crf6w.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">82</span>.96.35.7
www.spx5a4e5eyhkdtpt2xj.com
www.6phyovjhggkfm.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">83</span>.140.59.2
www.o5qzqtbs.com
www.bnymkm3nk7jtz3.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">82</span>.96.35.8
www.7wdf4rkj5mew.com
www.sd5mkmsmo.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">93</span>.180.156.45
www.rxy4jiw4wk.com
www.g66mipkcyhjwumywk4h.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">81</span>.218.109.195
www.gempmzrnwnk.com
www.6lrz7wtwprz.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">31</span>.172.30.4
www.4jvdpoo5wcklhd3usu.com
www.f4uxyorx2h.net
Source IP: <span class="m">10</span>.0.0.126
Destination IP: <span class="m">50</span>.7.194.122
www.pxznjv3t75.com
www.wuqq77l634eogfm.net
</pre></div>
<p>Please leave a comment if you have any questions.</p>Detecting Tor network traffic with SiLK2014-01-09T04:33:00-05:002014-01-09T04:33:00-05:00Stephen Reesetag:www.rsreese.com,2014-01-09:/detecting-tor-network-traffic-with-silk/This entry continues a series of posts on identifying Tor network traffic and usage. This post is not to argue the merits of allowing Tor to run on a network. However, the entry will demonstrate how to create a set of Tor server IP addresses to parse network flow using …<p>This entry continues a series of <a href="http://www.rsreese.com/tag/tor/">posts</a> on identifying Tor network
traffic and usage. This post is not to argue the merits of allowing Tor
to run on a network. However, the entry will demonstrate how to create a
set of Tor server <span class="caps">IP</span> addresses to parse network flow using SiLK (System
for Internet-Level Knowledge) in order to determine if the network flow
is a match. It is assumed you have downloaded, compiled and installed
<a href="http://tools.netsa.cert.org/silk/">SiLK</a>, <a href="http://tools.netsa.cert.org/yaf/">YaF</a>, and <a href="https://tools.netsa.cert.org/fixbuf/">libfixbuf</a>. Please see prior <a href="http://www.rsreese.com/tag/silk/">posts</a> on
this topic or the respective documentation for installation help if needed.</p>
<p>We need to obtain the current list of Tor servers and place them in a
file. We will then parse the destination <span class="caps">IP</span> addresses which will be
placed into a SiLK set using the SiLK <code>rwsetbuild</code> command. Creating an
<span class="caps">IP</span> set will allow us to use <em>rwfilter</em> to specify what <span class="caps">IP</span> addresses
should match outgoing network traffic. A Perl script from <a href="http://blog.vorant.com/2008/06/tor-server-lists-revisited.html">here</a> makes
quick work of downloading the current Tor server list.</p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/perl</span>
<span class="c1">#</span>
<span class="c1"># Fetch the list of known Tor servers (from an existing Tor server) and</span>
<span class="c1"># display some of the basic info for each router.</span>
<span class="k">use</span> <span class="nn">LWP::Simple</span><span class="p">;</span>
<span class="c1"># Hostname of an existing Tor router. We use one of the directory authorities</span>
<span class="c1"># since that's pretty much what they're for.</span>
<span class="nv">$INITIAL_TOR_SERVER</span> <span class="o">=</span> <span class="s">"193.23.244.244"</span><span class="p">;</span> <span class="c1"># http://dannenberg.ccc.de/tor/status/all</span>
<span class="nv">$DIR_PORT</span> <span class="o">=</span> <span class="mi">80</span><span class="p">;</span>
<span class="c1"># Fetch the list of servers</span>
<span class="nv">$content</span> <span class="o">=</span> <span class="n">get</span><span class="p">(</span><span class="s">"http://$INITIAL_TOR_SERVER:$DIR_PORT/tor/status/all"</span><span class="p">);</span>
<span class="nv">@lines</span> <span class="o">=</span> <span class="nb">split</span> <span class="sr">/\n/</span><span class="p">,</span><span class="nv">$content</span><span class="p">;</span>
<span class="k">foreach</span> <span class="nv">$router</span> <span class="p">(</span><span class="nv">@lines</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="nv">$router</span> <span class="o">=~</span> <span class="sr">m/^r\s+(\S+)\s+(\S+)\s+(\S+)\s+(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+(\S+)\s+(\d+)\s+(\d+)$/</span><span class="p">)</span> <span class="p">{</span>
<span class="p">(</span><span class="nv">$name</span><span class="p">,</span> <span class="nv">$address</span><span class="p">,</span> <span class="nv">$or_port</span><span class="p">,</span> <span class="nv">$directory_port</span><span class="p">,</span> <span class="nv">$update_time</span><span class="p">)</span> <span class="o">=</span>
<span class="p">(</span><span class="nv">$1</span><span class="p">,</span> <span class="nv">$5</span><span class="p">,</span> <span class="nv">$6</span><span class="p">,</span> <span class="nv">$7</span><span class="p">,</span> <span class="nv">$4</span><span class="p">);</span>
<span class="k">print</span> <span class="s">"$name | $address | $or_port | $directory_port | $update_time\n"</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Now that we have the current Tor server list, we can parse the Tor <span class="caps">IP</span>
addresses. While you can modify the Perl script to only display the Tor
server <span class="caps">IP</span> addresses, I still like to sort and parse for unique addresses
as there are could be duplicates. You could also specify what type of
Tor <span class="caps">IP</span> addresses you would like, i.e. exit, active, etc. Further, it is
not bad to have a reference to determine what ports are associated with
which addresses. Useful for more advanced queries.</p>
<div class="highlight"><pre><span></span>$ awk -F <span class="s2">"|"</span> <span class="s1">'{ print $2 }'</span> exit-addresses <span class="p">|</span> awk <span class="s1">'{sub(/^[ \t]+/, "")};1'</span> <span class="p">|</span>sort<span class="p">|</span>uniq > tor.txt
</pre></div>
<p>We convert the file containing the Tor server <span class="caps">IP</span> addresses to a set
using the following command:</p>
<div class="highlight"><pre><span></span>$ rwsetbuild tor.txt tor-servers.set
</pre></div>
<p>Typically, network flow would have already been captured for
retrospective analysis, but for example sake, we will use a packet
capture which already contains Tor traffic. We first convert our
captured traffic to a YaF formatted file. This example <span class="caps">PCAP</span> may be
downloaded from <a href="http://www.cloudshark.org/captures/96ed6d98c159">CloudShark</a>.</p>
<div class="highlight"><pre><span></span>$ /usr/local/bin/yaf --in tor.pcap --out ~/tor.yaf --filter<span class="o">=</span><span class="s2">"port 443"</span> --applabel --applabel-rules<span class="o">=</span>/usr/local/etc/yafApplabelRules.conf --max-payload<span class="o">=</span><span class="m">4000</span> --plugin-name<span class="o">=</span>/usr/local/lib/yaf/dpacketplugin.la --plugin-opts<span class="o">=</span><span class="s2">"443"</span> --lock <span class="p">&</span>
</pre></div>
<p>Next, we convert the YaF format file to an <span class="caps">IPFIX</span> formatted file.</p>
<div class="highlight"><pre><span></span>$ rwipfix2silk --silk-output<span class="o">=</span>tor.rw tor.yaf
</pre></div>
<p>This <em>rwfilter</em> query parses for the data we are looking for and places
in a binary file. We can write to standard out but I usually end up
running additional queries using tools such as <em>rwcut</em> and <em>rwstats</em> so
it is much faster to work from the smaller binary file, verse running
the original query again.</p>
<div class="highlight"><pre><span></span>$ rwfilter --start-date<span class="o">=</span><span class="m">2013</span>/12/30 --end-date<span class="o">=</span><span class="m">2013</span>/12/30 --dipset<span class="o">=</span>tor-servers.set --proto<span class="o">=</span><span class="m">0</span>- --type<span class="o">=</span>all --pass<span class="o">=</span>tor2.bin tor.rw
</pre></div>
<p>We parse the SiLK records we are interested in seeing to standard out
via the <em>rwcut</em> command. Note the use of the <em>cut</em> command to minimize
the white-space prefixing the output.</p>
<div class="highlight"><pre><span></span>$ rwcut tor2.bin<span class="p">|</span>cut -c26-
sIP<span class="p">|</span> dIP<span class="p">|</span>sPort<span class="p">|</span>dPort<span class="p">|</span>pro<span class="p">|</span> packets<span class="p">|</span> bytes<span class="p">|</span> flags<span class="p">|</span> sTime<span class="p">|</span> duration<span class="p">|</span> eTime<span class="p">|</span>sen<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">198</span>.27.97.223<span class="p">|</span><span class="m">38946</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8497</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.336<span class="p">|</span> <span class="m">76</span>.182<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.518<span class="p">|</span> S0<span class="p">|</span>
<span class="m">198</span>.27.97.223<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">38946</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">32</span><span class="p">|</span> <span class="m">28802</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.381<span class="p">|</span> <span class="m">76</span>.137<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.518<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">96</span>.127.153.58<span class="p">|</span><span class="m">42529</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">27</span><span class="p">|</span> <span class="m">8341</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.190<span class="p">|</span> <span class="m">75</span>.341<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.531<span class="p">|</span> S0<span class="p">|</span>
<span class="m">96</span>.127.153.58<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">42529</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">26678</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.232<span class="p">|</span> <span class="m">75</span>.299<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.531<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">192</span>.151.147.5<span class="p">|</span><span class="m">44384</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">14</span><span class="p">|</span> <span class="m">3502</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:26.486<span class="p">|</span> <span class="m">71</span>.052<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.538<span class="p">|</span> S0<span class="p">|</span>
<span class="m">192</span>.151.147.5<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">44384</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">14</span><span class="p">|</span> <span class="m">4819</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:26.535<span class="p">|</span> <span class="m">71</span>.003<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.538<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">66</span>.18.12.197<span class="p">|</span><span class="m">49341</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">28</span><span class="p">|</span> <span class="m">8475</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.426<span class="p">|</span> <span class="m">76</span>.125<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.551<span class="p">|</span> S0<span class="p">|</span>
<span class="m">66</span>.18.12.197<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">49341</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">26805</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.471<span class="p">|</span> <span class="m">76</span>.080<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.551<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">64</span>.62.249.222<span class="p">|</span><span class="m">40742</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8159</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.375<span class="p">|</span> <span class="m">76</span>.208<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.583<span class="p">|</span> S0<span class="p">|</span>
<span class="m">64</span>.62.249.222<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">40742</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">32</span><span class="p">|</span> <span class="m">28493</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.461<span class="p">|</span> <span class="m">76</span>.122<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.583<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">212</span>.83.158.173<span class="p">|</span><span class="m">40825</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">28</span><span class="p">|</span> <span class="m">8394</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.079<span class="p">|</span> <span class="m">75</span>.506<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.585<span class="p">|</span> S0<span class="p">|</span>
<span class="m">212</span>.83.158.173<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">40825</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">31</span><span class="p">|</span> <span class="m">28867</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.180<span class="p">|</span> <span class="m">75</span>.405<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.585<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">212</span>.83.155.250<span class="p">|</span><span class="m">55603</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8454</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.196<span class="p">|</span> <span class="m">75</span>.389<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.585<span class="p">|</span> S0<span class="p">|</span>
<span class="m">212</span>.83.155.250<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">55603</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">31</span><span class="p">|</span> <span class="m">27840</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.290<span class="p">|</span> <span class="m">75</span>.295<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.585<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">212</span>.83.140.45<span class="p">|</span><span class="m">46797</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8455</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.342<span class="p">|</span> <span class="m">76</span>.245<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.587<span class="p">|</span> S0<span class="p">|</span>
<span class="m">212</span>.83.140.45<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">46797</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">26648</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.439<span class="p">|</span> <span class="m">76</span>.148<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.587<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">212</span>.83.158.50<span class="p">|</span><span class="m">50935</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">31</span><span class="p">|</span> <span class="m">8567</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.396<span class="p">|</span> <span class="m">76</span>.191<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.587<span class="p">|</span> S0<span class="p">|</span>
<span class="m">212</span>.83.158.50<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">50935</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">26145</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.492<span class="p">|</span> <span class="m">76</span>.095<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.587<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">212</span>.83.158.40<span class="p">|</span><span class="m">33170</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8459</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.088<span class="p">|</span> <span class="m">75</span>.506<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.594<span class="p">|</span> S0<span class="p">|</span>
<span class="m">212</span>.83.158.40<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">33170</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">33</span><span class="p">|</span> <span class="m">28930</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:23.199<span class="p">|</span> <span class="m">74</span>.395<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.594<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">212</span>.83.158.5<span class="p">|</span><span class="m">37960</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">27</span><span class="p">|</span> <span class="m">8342</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.415<span class="p">|</span> <span class="m">76</span>.187<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.602<span class="p">|</span> S0<span class="p">|</span>
<span class="m">212</span>.83.158.5<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">37960</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">32</span><span class="p">|</span> <span class="m">26758</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.517<span class="p">|</span> <span class="m">76</span>.085<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.602<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">31</span>.7.186.228<span class="p">|</span><span class="m">44997</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">26</span><span class="p">|</span> <span class="m">8294</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.377<span class="p">|</span> <span class="m">76</span>.227<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.604<span class="p">|</span> S0<span class="p">|</span>
<span class="m">31</span>.7.186.228<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">44997</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">34</span><span class="p">|</span> <span class="m">29440</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.486<span class="p">|</span> <span class="m">76</span>.118<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.604<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">216</span>.66.85.146<span class="p">|</span><span class="m">50817</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">15</span><span class="p">|</span> <span class="m">3379</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:21:34.492<span class="p">|</span> <span class="m">3</span>.114<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.606<span class="p">|</span> S0<span class="p">|</span>
<span class="m">216</span>.66.85.146<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">50817</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">15</span><span class="p">|</span> <span class="m">6866</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:21:34.590<span class="p">|</span> <span class="m">3</span>.016<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.606<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">178</span>.254.35.132<span class="p">|</span><span class="m">50724</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">20</span><span class="p">|</span> <span class="m">5347</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:33.494<span class="p">|</span> <span class="m">64</span>.117<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.611<span class="p">|</span> S0<span class="p">|</span>
<span class="m">178</span>.254.35.132<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">50724</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">23</span><span class="p">|</span> <span class="m">16358</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:33.595<span class="p">|</span> <span class="m">64</span>.016<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.611<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">188</span>.40.98.96<span class="p">|</span><span class="m">54796</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8565</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.380<span class="p">|</span> <span class="m">76</span>.231<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.611<span class="p">|</span> S0<span class="p">|</span>
<span class="m">188</span>.40.98.96<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">54796</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">32</span><span class="p">|</span> <span class="m">27966</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.494<span class="p">|</span> <span class="m">76</span>.117<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.611<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">80</span>.100.45.156<span class="p">|</span><span class="m">60680</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8578</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.386<span class="p">|</span> <span class="m">76</span>.228<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.614<span class="p">|</span> S0<span class="p">|</span>
<span class="m">80</span>.100.45.156<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">60680</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">31</span><span class="p">|</span> <span class="m">28447</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.496<span class="p">|</span> <span class="m">76</span>.118<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.614<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">91</span>.143.91.174<span class="p">|</span><span class="m">39275</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">23</span><span class="p">|</span> <span class="m">8209</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.185<span class="p">|</span> <span class="m">75</span>.435<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.620<span class="p">|</span> S0<span class="p">|</span>
<span class="m">91</span>.143.91.174<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">39275</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">33</span><span class="p">|</span> <span class="m">28626</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.312<span class="p">|</span> <span class="m">75</span>.308<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.620<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">85</span>.17.122.80<span class="p">|</span><span class="m">43989</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8457</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.418<span class="p">|</span> <span class="m">76</span>.202<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.620<span class="p">|</span> S0<span class="p">|</span>
<span class="m">85</span>.17.122.80<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">43989</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">32</span><span class="p">|</span> <span class="m">28409</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.539<span class="p">|</span> <span class="m">76</span>.081<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.620<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">88</span>.159.20.120<span class="p">|</span><span class="m">49609</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">31</span><span class="p">|</span> <span class="m">8633</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.412<span class="p">|</span> <span class="m">76</span>.208<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.620<span class="p">|</span> S0<span class="p">|</span>
<span class="m">88</span>.159.20.120<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">49609</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">34</span><span class="p">|</span> <span class="m">29194</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.513<span class="p">|</span> <span class="m">76</span>.107<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.620<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">37</span>.59.150.178<span class="p">|</span><span class="m">47658</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8516</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.399<span class="p">|</span> <span class="m">76</span>.223<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.622<span class="p">|</span> S0<span class="p">|</span>
<span class="m">37</span>.59.150.178<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">47658</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">33</span><span class="p">|</span> <span class="m">29412</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.513<span class="p">|</span> <span class="m">76</span>.109<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.622<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">91</span>.219.237.229<span class="p">|</span><span class="m">35498</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">15</span><span class="p">|</span> <span class="m">3616</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:21:34.489<span class="p">|</span> <span class="m">3</span>.134<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.623<span class="p">|</span> S0<span class="p">|</span>
<span class="m">91</span>.219.237.229<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">35498</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">14</span><span class="p">|</span> <span class="m">7664</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:21:34.614<span class="p">|</span> <span class="m">3</span>.009<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.623<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">95</span>.211.225.167<span class="p">|</span><span class="m">57656</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">27</span><span class="p">|</span> <span class="m">8359</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.345<span class="p">|</span> <span class="m">76</span>.280<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.625<span class="p">|</span> S0<span class="p">|</span>
<span class="m">95</span>.211.225.167<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">57656</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">33</span><span class="p">|</span> <span class="m">27948</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.475<span class="p">|</span> <span class="m">76</span>.150<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.625<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">82</span>.96.35.7<span class="p">|</span><span class="m">58655</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">15</span><span class="p">|</span> <span class="m">3563</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:21:34.486<span class="p">|</span> <span class="m">3</span>.147<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.633<span class="p">|</span> S0<span class="p">|</span>
<span class="m">82</span>.96.35.7<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">58655</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">13</span><span class="p">|</span> <span class="m">7445</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:21:34.629<span class="p">|</span> <span class="m">3</span>.004<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.633<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">83</span>.140.59.2<span class="p">|</span><span class="m">45720</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">22</span><span class="p">|</span> <span class="m">8160</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.745<span class="p">|</span> <span class="m">75</span>.888<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.633<span class="p">|</span> S0<span class="p">|</span>
<span class="m">83</span>.140.59.2<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">45720</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">27422</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.887<span class="p">|</span> <span class="m">75</span>.746<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.633<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">82</span>.96.35.8<span class="p">|</span><span class="m">42995</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">28</span><span class="p">|</span> <span class="m">8414</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.339<span class="p">|</span> <span class="m">76</span>.302<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.641<span class="p">|</span> S0<span class="p">|</span>
<span class="m">82</span>.96.35.8<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">42995</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">33</span><span class="p">|</span> <span class="m">28927</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.479<span class="p">|</span> <span class="m">76</span>.162<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.641<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">93</span>.180.156.45<span class="p">|</span><span class="m">47282</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">33</span><span class="p">|</span> <span class="m">8671</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.421<span class="p">|</span> <span class="m">76</span>.223<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.644<span class="p">|</span> S0<span class="p">|</span>
<span class="m">93</span>.180.156.45<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">47282</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">39</span><span class="p">|</span> <span class="m">31370</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.562<span class="p">|</span> <span class="m">76</span>.082<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.644<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">81</span>.218.109.195<span class="p">|</span><span class="m">60000</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8460</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.383<span class="p">|</span> <span class="m">76</span>.277<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.660<span class="p">|</span> S0<span class="p">|</span>
<span class="m">81</span>.218.109.195<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">60000</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">32</span><span class="p">|</span> <span class="m">27852</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.535<span class="p">|</span> <span class="m">76</span>.125<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.660<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">31</span>.172.30.4<span class="p">|</span><span class="m">35914</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">36</span><span class="p">|</span> <span class="m">8922</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.146<span class="p">|</span> <span class="m">75</span>.538<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.684<span class="p">|</span> S0<span class="p">|</span>
<span class="m">31</span>.172.30.4<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">35914</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">34</span><span class="p">|</span> <span class="m">32082</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.271<span class="p">|</span> <span class="m">75</span>.413<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.684<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">50</span>.7.194.122<span class="p">|</span><span class="m">38522</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">20</span><span class="p">|</span> <span class="m">5384</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:33.487<span class="p">|</span> <span class="m">64</span>.202<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.689<span class="p">|</span> S0<span class="p">|</span>
<span class="m">50</span>.7.194.122<span class="p">|</span> <span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">443</span><span class="p">|</span><span class="m">38522</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">17</span><span class="p">|</span> <span class="m">9223</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:33.671<span class="p">|</span> <span class="m">64</span>.018<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.689<span class="p">|</span> S0<span class="p">|</span>
</pre></div>
<p>With the next query, we adjust the <em>type</em> of traffic we want to look at
to only outgoing traffic to the Tor servers instead of the previously
displayed bi-directional traffic.</p>
<div class="highlight"><pre><span></span>$ rwfilter --dipset<span class="o">=</span>tor-servers.set --proto<span class="o">=</span><span class="m">0</span>- --type<span class="o">=</span>out --pass<span class="o">=</span>tor.bin tor.rw
</pre></div>
<p>Again, we parse the SiLK records. Again, note the use of the cut command
to minimize the white-space prefix the first column of data. The reason
for this is there are additional columns of data not displayed by
default. Checkout the <em>rwcut</em> man page for other columns data that may
be of interest.</p>
<div class="highlight"><pre><span></span>$ rwcut tor.bin <span class="p">|</span>cut -c30-
sIP<span class="p">|</span> dIP<span class="p">|</span>sPort<span class="p">|</span>dPort<span class="p">|</span>pro<span class="p">|</span> packets<span class="p">|</span> bytes<span class="p">|</span> flags<span class="p">|</span> sTime<span class="p">|</span> duration<span class="p">|</span> eTime<span class="p">|</span>sen<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">198</span>.27.97.223<span class="p">|</span><span class="m">38946</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8497</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.336<span class="p">|</span> <span class="m">76</span>.182<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.518<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">96</span>.127.153.58<span class="p">|</span><span class="m">42529</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">27</span><span class="p">|</span> <span class="m">8341</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.190<span class="p">|</span> <span class="m">75</span>.341<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.531<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">192</span>.151.147.5<span class="p">|</span><span class="m">44384</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">14</span><span class="p">|</span> <span class="m">3502</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:26.486<span class="p">|</span> <span class="m">71</span>.052<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.538<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">66</span>.18.12.197<span class="p">|</span><span class="m">49341</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">28</span><span class="p">|</span> <span class="m">8475</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.426<span class="p">|</span> <span class="m">76</span>.125<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.551<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">64</span>.62.249.222<span class="p">|</span><span class="m">40742</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8159</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.375<span class="p">|</span> <span class="m">76</span>.208<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.583<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">212</span>.83.158.173<span class="p">|</span><span class="m">40825</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">28</span><span class="p">|</span> <span class="m">8394</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.079<span class="p">|</span> <span class="m">75</span>.506<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.585<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">212</span>.83.155.250<span class="p">|</span><span class="m">55603</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8454</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.196<span class="p">|</span> <span class="m">75</span>.389<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.585<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">212</span>.83.140.45<span class="p">|</span><span class="m">46797</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8455</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.342<span class="p">|</span> <span class="m">76</span>.245<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.587<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">212</span>.83.158.50<span class="p">|</span><span class="m">50935</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">31</span><span class="p">|</span> <span class="m">8567</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.396<span class="p">|</span> <span class="m">76</span>.191<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.587<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">212</span>.83.158.40<span class="p">|</span><span class="m">33170</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8459</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.088<span class="p">|</span> <span class="m">75</span>.506<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.594<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">212</span>.83.158.5<span class="p">|</span><span class="m">37960</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">27</span><span class="p">|</span> <span class="m">8342</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.415<span class="p">|</span> <span class="m">76</span>.187<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.602<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">31</span>.7.186.228<span class="p">|</span><span class="m">44997</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">26</span><span class="p">|</span> <span class="m">8294</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.377<span class="p">|</span> <span class="m">76</span>.227<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.604<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">216</span>.66.85.146<span class="p">|</span><span class="m">50817</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">15</span><span class="p">|</span> <span class="m">3379</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:21:34.492<span class="p">|</span> <span class="m">3</span>.114<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.606<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">178</span>.254.35.132<span class="p">|</span><span class="m">50724</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">20</span><span class="p">|</span> <span class="m">5347</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:33.494<span class="p">|</span> <span class="m">64</span>.117<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.611<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">188</span>.40.98.96<span class="p">|</span><span class="m">54796</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8565</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.380<span class="p">|</span> <span class="m">76</span>.231<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.611<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">80</span>.100.45.156<span class="p">|</span><span class="m">60680</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8578</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.386<span class="p">|</span> <span class="m">76</span>.228<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.614<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">91</span>.143.91.174<span class="p">|</span><span class="m">39275</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">23</span><span class="p">|</span> <span class="m">8209</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.185<span class="p">|</span> <span class="m">75</span>.435<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.620<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">85</span>.17.122.80<span class="p">|</span><span class="m">43989</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8457</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.418<span class="p">|</span> <span class="m">76</span>.202<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.620<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">88</span>.159.20.120<span class="p">|</span><span class="m">49609</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">31</span><span class="p">|</span> <span class="m">8633</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.412<span class="p">|</span> <span class="m">76</span>.208<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.620<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">37</span>.59.150.178<span class="p">|</span><span class="m">47658</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8516</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.399<span class="p">|</span> <span class="m">76</span>.223<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.622<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">91</span>.219.237.229<span class="p">|</span><span class="m">35498</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">15</span><span class="p">|</span> <span class="m">3616</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:21:34.489<span class="p">|</span> <span class="m">3</span>.134<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.623<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">95</span>.211.225.167<span class="p">|</span><span class="m">57656</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">27</span><span class="p">|</span> <span class="m">8359</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.345<span class="p">|</span> <span class="m">76</span>.280<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.625<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">82</span>.96.35.7<span class="p">|</span><span class="m">58655</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">15</span><span class="p">|</span> <span class="m">3563</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:21:34.486<span class="p">|</span> <span class="m">3</span>.147<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.633<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">83</span>.140.59.2<span class="p">|</span><span class="m">45720</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">22</span><span class="p">|</span> <span class="m">8160</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.745<span class="p">|</span> <span class="m">75</span>.888<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.633<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">82</span>.96.35.8<span class="p">|</span><span class="m">42995</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">28</span><span class="p">|</span> <span class="m">8414</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.339<span class="p">|</span> <span class="m">76</span>.302<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.641<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">93</span>.180.156.45<span class="p">|</span><span class="m">47282</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">33</span><span class="p">|</span> <span class="m">8671</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.421<span class="p">|</span> <span class="m">76</span>.223<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.644<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">81</span>.218.109.195<span class="p">|</span><span class="m">60000</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8460</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.383<span class="p">|</span> <span class="m">76</span>.277<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.660<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">31</span>.172.30.4<span class="p">|</span><span class="m">35914</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">36</span><span class="p">|</span> <span class="m">8922</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.146<span class="p">|</span> <span class="m">75</span>.538<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.684<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">50</span>.7.194.122<span class="p">|</span><span class="m">38522</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">20</span><span class="p">|</span> <span class="m">5384</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:33.487<span class="p">|</span> <span class="m">64</span>.202<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.689<span class="p">|</span> S0<span class="p">|</span>
</pre></div>
<p>Lastly, we take a look at the reverse entries. As you can see, it is
apparent that some of the hosts have Tor tertiary domain names which
suggests that some of the flows may be destined for Tor servers.</p>
<div class="highlight"><pre><span></span>$ rwcut tor.bin <span class="p">|</span>rwresolve <span class="p">|</span>cut -c30-
sIP<span class="p">|</span> dIP<span class="p">|</span>sPort<span class="p">|</span>dPort<span class="p">|</span>pro<span class="p">|</span> packets<span class="p">|</span> bytes<span class="p">|</span> flags<span class="p">|</span> sTime<span class="p">|</span> duration<span class="p">|</span> eTime<span class="p">|</span>sen<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span><span class="m">198</span>.27.97.223.vpsrealm.com<span class="p">|</span><span class="m">38946</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8497</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.336<span class="p">|</span> <span class="m">76</span>.182<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.518<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>xxviii.example.tld<span class="p">|</span><span class="m">42529</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">27</span><span class="p">|</span> <span class="m">8341</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.190<span class="p">|</span> <span class="m">75</span>.341<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.531<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>tor.koehn.com<span class="p">|</span><span class="m">44384</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">14</span><span class="p">|</span> <span class="m">3502</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:26.486<span class="p">|</span> <span class="m">71</span>.052<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.538<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">66</span>.18.12.197<span class="p">|</span><span class="m">49341</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">28</span><span class="p">|</span> <span class="m">8475</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.426<span class="p">|</span> <span class="m">76</span>.125<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.551<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>hecustomer.10gigabitethernet8-1.core1.pao1.he.net<span class="p">|</span><span class="m">40742</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8159</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.375<span class="p">|</span> <span class="m">76</span>.208<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.583<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>n5.servbr.net<span class="p">|</span><span class="m">40825</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">28</span><span class="p">|</span> <span class="m">8394</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.079<span class="p">|</span> <span class="m">75</span>.506<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.585<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>n15.servbr.net<span class="p">|</span><span class="m">55603</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8454</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.196<span class="p">|</span> <span class="m">75</span>.389<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.585<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span><span class="m">212</span>-83-140-45.rev.poneytelecom.eu<span class="p">|</span><span class="m">46797</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8455</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.342<span class="p">|</span> <span class="m">76</span>.245<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.587<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>n13.servbr.net<span class="p">|</span><span class="m">50935</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">31</span><span class="p">|</span> <span class="m">8567</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.396<span class="p">|</span> <span class="m">76</span>.191<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.587<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>n12.servbr.net<span class="p">|</span><span class="m">33170</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8459</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.088<span class="p">|</span> <span class="m">75</span>.506<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.594<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>n10.servbr.net<span class="p">|</span><span class="m">37960</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">27</span><span class="p">|</span> <span class="m">8342</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.415<span class="p">|</span> <span class="m">76</span>.187<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.602<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">31</span>.7.186.228<span class="p">|</span><span class="m">44997</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">26</span><span class="p">|</span> <span class="m">8294</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.377<span class="p">|</span> <span class="m">76</span>.227<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.604<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>hecustomer.10gigabitethernet1-2.core1.ams1.he.net<span class="p">|</span><span class="m">50817</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">15</span><span class="p">|</span> <span class="m">3379</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:21:34.492<span class="p">|</span> <span class="m">3</span>.114<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.606<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>v37433.1blu.de<span class="p">|</span><span class="m">50724</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">20</span><span class="p">|</span> <span class="m">5347</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:33.494<span class="p">|</span> <span class="m">64</span>.117<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.611<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>static.188-40-98-96.clients.your-server.de<span class="p">|</span><span class="m">54796</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8565</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.380<span class="p">|</span> <span class="m">76</span>.231<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.611<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>a80-100-45-156.adsl.xs4all.nl<span class="p">|</span><span class="m">60680</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8578</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.386<span class="p">|</span> <span class="m">76</span>.228<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.614<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span><span class="m">91</span>.143.91.174<span class="p">|</span><span class="m">39275</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">23</span><span class="p">|</span> <span class="m">8209</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.185<span class="p">|</span> <span class="m">75</span>.435<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.620<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span> <span class="m">85</span>.17.122.80<span class="p">|</span><span class="m">43989</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8457</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.418<span class="p">|</span> <span class="m">76</span>.202<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.620<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span><span class="m">120</span>-20-159-88.business.edutel.nl<span class="p">|</span><span class="m">49609</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">31</span><span class="p">|</span> <span class="m">8633</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.412<span class="p">|</span> <span class="m">76</span>.208<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.620<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span><span class="m">37</span>-59-150-178.static-ip.hostplanet.me<span class="p">|</span><span class="m">47658</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">30</span><span class="p">|</span> <span class="m">8516</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.399<span class="p">|</span> <span class="m">76</span>.223<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.622<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>sa0111.azar-a.net<span class="p">|</span><span class="m">35498</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">15</span><span class="p">|</span> <span class="m">3616</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:21:34.489<span class="p">|</span> <span class="m">3</span>.134<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.623<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>greendale.badexample.net<span class="p">|</span><span class="m">57656</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">27</span><span class="p">|</span> <span class="m">8359</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.345<span class="p">|</span> <span class="m">76</span>.280<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.625<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>luftgitarr.mooo.se<span class="p">|</span><span class="m">58655</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">15</span><span class="p">|</span> <span class="m">3563</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:21:34.486<span class="p">|</span> <span class="m">3</span>.147<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.633<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>kimya.mooo.se<span class="p">|</span><span class="m">45720</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">22</span><span class="p">|</span> <span class="m">8160</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.745<span class="p">|</span> <span class="m">75</span>.888<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.633<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>junis.mooo.se<span class="p">|</span><span class="m">42995</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">28</span><span class="p">|</span> <span class="m">8414</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.339<span class="p">|</span> <span class="m">76</span>.302<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.641<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>tor.b0red.de<span class="p">|</span><span class="m">47282</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">33</span><span class="p">|</span> <span class="m">8671</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.421<span class="p">|</span> <span class="m">76</span>.223<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.644<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span><span class="m">195</span>.ab4.interhost.co.il<span class="p">|</span><span class="m">60000</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">29</span><span class="p">|</span> <span class="m">8460</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:21.383<span class="p">|</span> <span class="m">76</span>.277<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.660<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>tor21.anonymizer.ccc.de<span class="p">|</span><span class="m">35914</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">36</span><span class="p">|</span> <span class="m">8922</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:22.146<span class="p">|</span> <span class="m">75</span>.538<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.684<span class="p">|</span> S0<span class="p">|</span>
<span class="m">10</span>.0.0.126<span class="p">|</span>torsrvl.snydernet.net<span class="p">|</span><span class="m">38522</span><span class="p">|</span> <span class="m">443</span><span class="p">|</span> <span class="m">6</span><span class="p">|</span> <span class="m">20</span><span class="p">|</span> <span class="m">5384</span><span class="p">|</span>FS PA <span class="p">|</span><span class="m">2013</span>/12/30T20:20:33.487<span class="p">|</span> <span class="m">64</span>.202<span class="p">|</span><span class="m">2013</span>/12/30T20:21:37.689<span class="p">|</span> S0<span class="p">|</span>
</pre></div>
<p>Or we can use the <em>rwuniq</em> command to list the unique destinations,
again piping through <em>rwresolve</em>:</p>
<div class="highlight"><pre><span></span>$ rwuniq --fields<span class="o">=</span><span class="m">2</span> --no-columns tor.bin <span class="p">|</span>rwresolve
dIP<span class="p">|</span>Records<span class="p">|</span>
luftgitarr.mooo.se<span class="p">|</span><span class="m">1</span><span class="p">|</span>
tor.b0red.de<span class="p">|</span><span class="m">1</span><span class="p">|</span>
junis.mooo.se<span class="p">|</span><span class="m">1</span><span class="p">|</span>
<span class="m">31</span>.7.186.228<span class="p">|</span><span class="m">1</span><span class="p">|</span>
tor21.anonymizer.ccc.de<span class="p">|</span><span class="m">1</span><span class="p">|</span>
xxviii.example.tld<span class="p">|</span><span class="m">1</span><span class="p">|</span>
tor.koehn.com<span class="p">|</span><span class="m">1</span><span class="p">|</span>
n15.servbr.net<span class="p">|</span><span class="m">1</span><span class="p">|</span>
a80-100-45-156.adsl.xs4all.nl<span class="p">|</span><span class="m">1</span><span class="p">|</span>
n13.servbr.net<span class="p">|</span><span class="m">1</span><span class="p">|</span>
<span class="m">120</span>-20-159-88.business.edutel.nl<span class="p">|</span><span class="m">1</span><span class="p">|</span>
<span class="m">91</span>.143.91.174<span class="p">|</span><span class="m">1</span><span class="p">|</span>
<span class="m">195</span>.ab4.interhost.co.il<span class="p">|</span><span class="m">1</span><span class="p">|</span>
<span class="m">37</span>-59-150-178.static-ip.hostplanet.me<span class="p">|</span><span class="m">1</span><span class="p">|</span>
sa0111.azar-a.net<span class="p">|</span><span class="m">1</span><span class="p">|</span>
static.188-40-98-96.clients.your-server.de<span class="p">|</span><span class="m">1</span><span class="p">|</span>
n5.servbr.net<span class="p">|</span><span class="m">1</span><span class="p">|</span>
torsrvl.snydernet.net<span class="p">|</span><span class="m">1</span><span class="p">|</span>
<span class="m">198</span>.27.97.223.vpsrealm.com<span class="p">|</span><span class="m">1</span><span class="p">|</span>
<span class="m">66</span>.18.12.197<span class="p">|</span><span class="m">1</span><span class="p">|</span>
v37433.1blu.de<span class="p">|</span><span class="m">1</span><span class="p">|</span>
hecustomer.10gigabitethernet1-2.core1.ams1.he.net<span class="p">|</span><span class="m">1</span><span class="p">|</span>
<span class="m">212</span>-83-140-45.rev.poneytelecom.eu<span class="p">|</span><span class="m">1</span><span class="p">|</span>
kimya.mooo.se<span class="p">|</span><span class="m">1</span><span class="p">|</span>
<span class="m">85</span>.17.122.80<span class="p">|</span><span class="m">1</span><span class="p">|</span>
n12.servbr.net<span class="p">|</span><span class="m">1</span><span class="p">|</span>
greendale.badexample.net<span class="p">|</span><span class="m">1</span><span class="p">|</span>
n10.servbr.net<span class="p">|</span><span class="m">1</span><span class="p">|</span>
hecustomer.10gigabitethernet8-1.core1.pao1.he.net<span class="p">|</span><span class="m">1</span><span class="p">|</span>
</pre></div>
<p>In conclusion, using SiLK we can provide retrospective analysis to
determine if traffic may be destined for Tor servers. While not a
definitive method of detection as there could be false-positives due to
hosting of legitimate services on Tor servers, it is a quick method to
get some insight. As usual, please leave a comment below if you have any
questions or comments.</p>Resizing Xen guest parition based filesystems2013-07-03T14:27:00-04:002013-07-03T14:27:00-04:00Stephen Reesetag:www.rsreese.com,2013-07-03:/resizing-xen-guest-parition-based-filesystems/This post assumes you are running the Xen hypervisor and are using a partitions based filesystems for you Xen guest you would like to re-size. I have previously written on Installing Xen on CentOS 6 from source and another blog entry that describes how to create partition based Xen guests …<p>This post assumes you are running the Xen hypervisor and are using a
partitions based filesystems for you Xen guest you would like to
re-size. I have previously written on <a href="http://www.rsreese.com/installing-xen-on-centos-6-from-source/" title="Installing Xen on CentOS 6 from source">Installing Xen on CentOS 6 from
source</a> and another blog entry that describes how to create partition
based Xen guests on <a href="http://www.rsreese.com/creating-debian-guests-on-xen-using-parition-based-filesystem/" title="Creating Debian guests on Xen using partition based filesystem">Creating Debian guests on Xen using partition based
filesystem</a> if you would like to see how to get started running Xen.</p>
<p>To resize, first shutdown the guest instance:</p>
<div class="highlight"><pre><span></span>$ sudo xm shutdown Wheezy
$ sudo lvresize /dev/VolGroup00/Wheezy -L +10GB
Extending logical volume Wheezy to <span class="m">20</span>.00 GiB
Logical volume Wheezy successfully resized
$ sudo lvdisplay
--- Logical volume ---
LV Path /dev/VolGroup00/Wheezy
LV Name Wheezy
VG Name VolGroup00
LV UUID jQqEFZ-sd39-siY6-kqCZ-l8Lq-UWWk-3f4oh5
LV Write Access read/write
LV Creation host, <span class="nb">time</span> host.localdomain, <span class="m">2013</span>-05-14 <span class="m">12</span>:32:00 -0400
LV Status available
<span class="c1"># open 0</span>
LV Size <span class="m">20</span>.00 GiB
Current LE <span class="m">5120</span>
Segments <span class="m">1</span>
Allocation inherit
Read ahead sectors auto
- currently <span class="nb">set</span> to <span class="m">256</span>
Block device <span class="m">253</span>:0
</pre></div>
<p>I would first backup the partition that is going to be modified. This is
going to sound weird; but this process uses fdisk to delete and recreate
the partition.</p>
<p>List you partition:</p>
<div class="highlight"><pre><span></span>$ sudo fdisk -l /dev/VolGroup00/Wheezy
Disk /dev/VolGroup00/Wheezy: <span class="m">21</span>.5 GB, <span class="m">21474836480</span> bytes
<span class="m">255</span> heads, <span class="m">63</span> sectors/track, <span class="m">2610</span> cylinders
<span class="nv">Units</span> <span class="o">=</span> cylinders of <span class="m">16065</span> * <span class="nv">512</span> <span class="o">=</span> <span class="m">8225280</span> bytes
Sector size <span class="o">(</span>logical/physical<span class="o">)</span>: <span class="m">512</span> bytes / <span class="m">512</span> bytes
I/O size <span class="o">(</span>minimum/optimal<span class="o">)</span>: <span class="m">512</span> bytes / <span class="m">512</span> bytes
Disk identifier: 0x00081c29
Device Boot Start End Blocks Id System
/dev/VolGroup00/Wheezy1 <span class="m">1</span> <span class="m">63</span> <span class="m">498688</span> <span class="m">82</span> Linux swap / Solaris
Partition <span class="m">1</span> does not end on cylinder boundary.
/dev/VolGroup00/Wheezy2 <span class="m">63</span> <span class="m">1306</span> <span class="m">9985024</span> <span class="m">83</span> Linux
</pre></div>
<p>When trying to directly re-size, an error occurs.</p>
<div class="highlight"><pre><span></span>$ sudo resize2fs /dev/VolGroup00/Wheezy
resize2fs <span class="m">1</span>.41.12 <span class="o">(</span><span class="m">17</span>-May-2010<span class="o">)</span>
resize2fs: Bad magic number in super-block <span class="k">while</span> trying to open /dev/VolGroup00/Wheezy
Couldn<span class="err">'</span>t find valid filesystem superblock.
</pre></div>
<p>We are now going to delete the partition, as warned before, make sure
you have backups.</p>
<div class="highlight"><pre><span></span>$ sudo fdisk /dev/VolGroup00/Wheezy
WARNING: DOS-compatible mode is deprecated. It<span class="s1">'s strongly recommended to</span>
<span class="s1">switch off the mode (command '</span>c<span class="s1">') and change display units to</span>
<span class="s1">sectors (command '</span>u<span class="err">'</span><span class="o">)</span>.
Command <span class="o">(</span>m <span class="k">for</span> <span class="nb">help</span><span class="o">)</span>: p
Disk /dev/VolGroup00/Wheezy: <span class="m">21</span>.5 GB, <span class="m">21474836480</span> bytes
<span class="m">255</span> heads, <span class="m">63</span> sectors/track, <span class="m">2610</span> cylinders
<span class="nv">Units</span> <span class="o">=</span> cylinders of <span class="m">16065</span> * <span class="nv">512</span> <span class="o">=</span> <span class="m">8225280</span> bytes
Sector size <span class="o">(</span>logical/physical<span class="o">)</span>: <span class="m">512</span> bytes / <span class="m">512</span> bytes
I/O size <span class="o">(</span>minimum/optimal<span class="o">)</span>: <span class="m">512</span> bytes / <span class="m">512</span> bytes
Disk identifier: 0x00081c29
Device Boot Start End Blocks Id System
/dev/VolGroup00/Wheezy1 <span class="m">1</span> <span class="m">63</span> <span class="m">498688</span> <span class="m">82</span> Linux swap / Solaris
Partition <span class="m">1</span> does not end on cylinder boundary.
/dev/VolGroup00/Wheezy2 <span class="m">63</span> <span class="m">1306</span> <span class="m">9985024</span> <span class="m">83</span> Linux
Command <span class="o">(</span>m <span class="k">for</span> <span class="nb">help</span><span class="o">)</span>: d
Partition number <span class="o">(</span><span class="m">1</span>-4<span class="o">)</span>: <span class="m">2</span>
Command <span class="o">(</span>m <span class="k">for</span> <span class="nb">help</span><span class="o">)</span>: p
Disk /dev/VolGroup00/Wheezy: <span class="m">21</span>.5 GB, <span class="m">21474836480</span> bytes
<span class="m">255</span> heads, <span class="m">63</span> sectors/track, <span class="m">2610</span> cylinders
<span class="nv">Units</span> <span class="o">=</span> cylinders of <span class="m">16065</span> * <span class="nv">512</span> <span class="o">=</span> <span class="m">8225280</span> bytes
Sector size <span class="o">(</span>logical/physical<span class="o">)</span>: <span class="m">512</span> bytes / <span class="m">512</span> bytes
I/O size <span class="o">(</span>minimum/optimal<span class="o">)</span>: <span class="m">512</span> bytes / <span class="m">512</span> bytes
Disk identifier: 0x00081c29
Device Boot Start End Blocks Id System
/dev/VolGroup00/Wheezy1 <span class="m">1</span> <span class="m">63</span> <span class="m">498688</span> <span class="m">82</span> Linux swap / Solaris
Partition <span class="m">1</span> does not end on cylinder boundary.
</pre></div>
<p>Recreate the partition with the new size.</p>
<div class="highlight"><pre><span></span>Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (63-2610, default 63):
Using default value 63
Last cylinder, +cylinders or +size{K,M,G} (63-2610, default 2610):
Using default value 2610
Command (m for help): p
Disk /dev/VolGroup00/Wheezy: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00081c29
Device Boot Start End Blocks Id System
/dev/VolGroup00/Wheezy1 1 63 498688 82 Linux swap / Solaris
Partition 1 does not end on cylinder boundary.
/dev/VolGroup00/Wheezy2 63 2610 20465113 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
</pre></div>
<p>The follow command splits the partitions apart as using the simple
Debian partitioning scheme may combine them.</p>
<div class="highlight"><pre><span></span>$ sudo kpartx -a /dev/VolGroup00/Wheezy
$ <span class="nb">cd</span> /dev/mapper/
$ ls
control VolGroup00-Wheezy VolGroup00-Wheezy1 VolGroup00-Wheezy2
</pre></div>
<p>Next, check the filesystem for errors.</p>
<div class="highlight"><pre><span></span>$ sudo e2fsck -f VolGroup00-Wheezy2
e2fsck <span class="m">1</span>.41.12 <span class="o">(</span><span class="m">17</span>-May-2010<span class="o">)</span>
Pass <span class="m">1</span>: Checking inodes, blocks, and sizes
Pass <span class="m">2</span>: Checking directory structure
Pass <span class="m">3</span>: Checking directory connectivity
Pass <span class="m">4</span>: Checking reference counts
Pass <span class="m">5</span>: Checking group summary information
VolGroup00-Wheezy2: <span class="m">29159</span>/624624 files <span class="o">(</span><span class="m">0</span>.2% non-contiguous<span class="o">)</span>, <span class="m">224352</span>/2496256 blocks
</pre></div>
<p>We can now re-size the filesystem.</p>
<div class="highlight"><pre><span></span>$ sudo resize2fs VolGroup00-Wheezy2
resize2fs <span class="m">1</span>.41.12 <span class="o">(</span><span class="m">17</span>-May-2010<span class="o">)</span>
Resizing the filesystem on VolGroup00-Wheezy2 to <span class="m">5116278</span> <span class="o">(</span>4k<span class="o">)</span> blocks.
The filesystem on VolGroup00-Wheezy2 is now <span class="m">5116278</span> blocks long.
</pre></div>
<p>Reattach the filesystems that were previously split.</p>
<div class="highlight"><pre><span></span>$ sudo kpartx -d /dev/VolGroup00/Wheezy
$ ls
control VolGroup00-Wheezy
</pre></div>
<p>A quick look at the logical volume and we can see we grew from 10 to 20 Gigabytes.</p>
<div class="highlight"><pre><span></span>$ sudo lvscan
ACTIVE <span class="s1">'/dev/VolGroup00/Wheezy'</span> <span class="o">[</span><span class="m">20</span>.00 GiB<span class="o">]</span> inherit
</pre></div>
<p>You should now be able to boot the guest using the larger file system.</p>
<p>To delete the guest filesystem:</p>
<div class="highlight"><pre><span></span>sudo vgremove lvmxen
sudo pvremove /dev/sdb1
sudo parted /dev/sdb
(parted) rm 1
(parted) quit
</pre></div>Creating Debian guests on Xen using parition based filesystem2013-06-29T17:47:00-04:002013-06-29T17:47:00-04:00Stephen Reesetag:www.rsreese.com,2013-06-29:/creating-debian-guests-on-xen-using-parition-based-filesystem/This guide describes how to create a filesystem and guest for the Xen hypervisor. This assumes you have a working Xen install with Dom U. I have described setting up a Xen hypervisor from source in another posted titled Installing Xen on CentOS 6 from source. Create a partition to …<p>This guide describes how to create a filesystem and guest for the <a href="http://www.xenproject.org/users/why-the-xen-project.html">Xen
hypervisor</a>. This assumes you have a working Xen install with Dom U. I
have described setting up a Xen hypervisor from source in another posted
titled <a href="http://www.rsreese.com/installing-xen-on-centos-6-from-source/" title="Installing Xen on CentOS 6 from source">Installing Xen on CentOS 6 from source</a>.</p>
<p>Create a partition to store virtual machines on. We want to use a
partition based verse file based file-system for our guests as the
performance is much better.</p>
<div class="highlight"><pre><span></span>$ sudo parted /dev/sdb
mklabel gpt
<span class="o">(</span>parted<span class="o">)</span> unit GB
<span class="o">(</span>parted<span class="o">)</span> mkpart VolGroup00 0GB 400GB
<span class="o">(</span>parted<span class="o">)</span> <span class="nb">set</span> <span class="m">1</span> lvm on
<span class="o">(</span>parted<span class="o">)</span> quit
<span class="o">(</span>parted<span class="o">)</span> p
Model: DELL PERC <span class="m">6</span>/i <span class="o">(</span>scsi<span class="o">)</span>
Disk /dev/sdb: 3999GB
Sector size <span class="o">(</span>logical/physical<span class="o">)</span>: 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
<span class="m">1</span> 1049kB 400GB 400GB VolGroup00 lvm
</pre></div>
<p>Create a partition for the first virtual machine.</p>
<div class="highlight"><pre><span></span>$ sudo pvcreate /dev/sdb1
$ sudo vgcreate VolGroup00 /dev/sdb1
$ sudo vgdisplay
--- Volume group ---
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas <span class="m">1</span>
Metadata Sequence No <span class="m">1</span>
VG Access read/write
VG Status resizable
MAX LV <span class="m">0</span>
Cur LV <span class="m">0</span>
Open LV <span class="m">0</span>
Max PV <span class="m">0</span>
Cur PV <span class="m">1</span>
Act PV <span class="m">1</span>
VG Size <span class="m">372</span>.53 GiB
PE Size <span class="m">4</span>.00 MiB
Total PE <span class="m">95367</span>
Alloc PE / Size <span class="m">0</span> / <span class="m">0</span>
Free PE / Size <span class="m">95367</span> / <span class="m">372</span>.53 GiB
VG UUID hdCkfh-twnj-Nu2V-FsTe-RsQg-PzlE-5w4QGR
</pre></div>
<p>Create a logical volume for the virtual machine.</p>
<div class="highlight"><pre><span></span>$ sudo lvcreate -L 10GB -n Wheezy VolGroup00
$ sudo lvdisplay
--- Logical volume ---
LV Path /dev/VolGroup00/Wheezy
LV Name Wheezy
VG Name VolGroup00
LV UUID jQqEFZ-sd39-siY6-kqCZ-l8Lq-UWWk-3f4oh5
LV Write Access read/write
LV Creation host, <span class="nb">time</span> host.localdomain, <span class="m">2013</span>-05-14 <span class="m">12</span>:32:00 -0400
LV Status available
<span class="c1"># open 0</span>
LV Size <span class="m">10</span>.00 GiB
Current LE <span class="m">2560</span>
Segments <span class="m">1</span>
Allocation inherit
Read ahead sectors auto
- currently <span class="nb">set</span> to <span class="m">256</span>
Block device <span class="m">253</span>:0
</pre></div>
<p>Get the latest Debian <a href="http://http.us.debian.org/debian/dists/wheezy/main/installer-i386/current/images/hd-media/">hd-media</a>. Specify these parameters in the
virtual machine configuration that will be used for the first start-up,
i.e. the install of your guest. A second configuration will be used for
booting the guest post-install.</p>
<div class="highlight"><pre><span></span>kernel = "/scratch/debian/wheezy/vmlinuz"
ramdisk = "/scratch/debian/wheezy/initrd.gz"
extra = "debian-installer/exit/always_halt=true -- console=hvc0"
memory = 512
name = "Wheezy"
vif = ['bridge=br0']
disk = ['phy:/dev/VolGroup00/Wheezy,xvda,w']
</pre></div>
<p>Connect to the new guest with a console and perform the installation.</p>
<div class="highlight"><pre><span></span>$ sudo xl create -c /etc/xen/install-debian.cfg
</pre></div>
<p>Start a guest without a console.</p>
<div class="highlight"><pre><span></span>$ sudo xl create /etc/xen/debian.cfg
</pre></div>
<p>Leave the console.</p>
<div class="highlight"><pre><span></span>$ <span class="s2">"Ctrl+]"</span>
</pre></div>
<p>List the instances.</p>
<div class="highlight"><pre><span></span>$ sudo xl list
Name ID Mem VCPUs State Time<span class="o">(</span>s<span class="o">)</span>
Domain-0 <span class="m">0</span> <span class="m">2048</span> <span class="m">1</span> r----- <span class="m">237</span>.4
Wheezy <span class="m">11</span> <span class="m">512</span> <span class="m">1</span> -b---- <span class="m">6</span>.8
</pre></div>
<p>Connect to the console.</p>
<div class="highlight"><pre><span></span>$ sudo xl console Wheezy
</pre></div>
<p>Leave the console.</p>
<div class="highlight"><pre><span></span>$ <span class="s2">"Ctrl+]"</span>
</pre></div>
<p>If you have any questions or feel something is missing, leave a comment below.</p>Installing Xen on CentOS 6 from source2013-06-29T17:33:00-04:002013-06-29T17:33:00-04:00Stephen Reesetag:www.rsreese.com,2013-06-29:/installing-xen-on-centos-6-from-source/I recently had a need to install Xen hypervisor on CentOS and most of the guides covered using the package maintainers version. Further, RHEL distributions favor using KVM. I did come across HowTo: Install XEN Dom0 on CentOS 6 from source but the domain was blocked (Google cache made quick …<p>I recently had a need to install Xen hypervisor on CentOS and most of
the guides covered using the package maintainers version. Further, <span class="caps">RHEL</span>
distributions favor using <span class="caps">KVM</span>. I did come across <a href="http://blog.tidyhosts.com/index.php/howto-install-xen-dom0-on-centos-6-from-source">HowTo: Install <span class="caps">XEN</span>
Dom0 on CentOS 6 from source</a> but the domain was blocked (Google cache
made quick work of getting around that issue) and there were a few steps
that felt unclear. I took that guide and made a few changes which are
reflected below. You may want to also reference the Xen <a href="http://wiki.xen.org/wiki/Compiling_Xen_From_Source">Wiki</a> CenOS
6.2, Xen 4.2.1, and Kernel version 3.9.2 were used in this example but
newer and older versions should be similar.</p>
<p>First install dependencies:</p>
<div class="highlight"><pre><span></span>yum groupinstall "Development Libraries"
yum groupinstall "Development Tools"
yum install transfig wget tar less texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel
yum install pciutils-libs pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial texinfo
yum install libidn-devel yajl yajl-devel ocaml ocaml-findlib ocaml-findlib-devel python-devel uuid-devel libuuid-devel openssl-devel
yum install glibc-devel.i686
yum install
libvirt python-virtinst
</pre></div>
<p>Download the latest Xen <a href="http://www.xenproject.org/downloads/xen-archives.html">source package</a>.</p>
<div class="highlight"><pre><span></span>$ tar xzf xen-4.2.1.tar.gz
$ <span class="nb">cd</span> xen-4.2.1
$ ./configure
$ make xen <span class="o">&&</span> make tools <span class="o">&&</span> make stubdom
$ sudo make install xen
$ sudo make install xen-tools
$ sudo make install stubdom
</pre></div>
<p>Prevent the screen from powering off:</p>
<div class="highlight"><pre><span></span>$ sudo sh -c <span class="s2">"echo '/usr/bin/setterm -powersave off' >> /etc/rc.local"</span>
</pre></div>
<p>Define the resources for domain 0:</p>
<div class="highlight"><pre><span></span>$ sudo sh -c <span class="s2">"echo 'xl sched-credit -d Domain-0 -w 512' >> /etc/xendom0caps"</span>
$ sudo chmod +x /etc/xendom0caps
</pre></div>
<p>Start the services at boot:</p>
<div class="highlight"><pre><span></span>sudo ln -s /etc/init.d/xendomains /etc/rc0.d/S10xendomains
sudo ln -s /etc/init.d/xendomains /etc/rc6.d/S10xendomains
sudo ln -s /etc/init.d/xendomains /etc/rc3.d/S98xendomains
sudo ln -s /etc/init.d/xencommons /etc/rc3.d/S98xencommons
sudo ln -s /etc/xendom0caps /etc/rc3.d/S97xendom0caps
</pre></div>
<p>Optionally for those that want to use the xm commands.</p>
<div class="highlight"><pre><span></span>sudo ln -s /etc/init.d/xend /etc/rc3.d/S98xend
</pre></div>
<p>Make sure everything is going to start at the correct runlevel. Note
that <strong>xend</strong> is optional</p>
<div class="highlight"><pre><span></span>$ chkconfig --list <span class="p">|</span>grep xen
xencommons <span class="m">0</span>:off <span class="m">1</span>:off <span class="m">2</span>:off <span class="m">3</span>:on <span class="m">4</span>:off <span class="m">5</span>:off <span class="m">6</span>:off
xend <span class="m">0</span>:off <span class="m">1</span>:off <span class="m">2</span>:off <span class="m">3</span>:on <span class="m">4</span>:off <span class="m">5</span>:off <span class="m">6</span>:off
xendomains <span class="m">0</span>:on <span class="m">1</span>:off <span class="m">2</span>:off <span class="m">3</span>:on <span class="m">4</span>:off <span class="m">5</span>:off <span class="m">6</span>:on
</pre></div>
<p>Make sure the weight is setup, this may vary depending your
needs/resources available.</p>
<div class="highlight"><pre><span></span>$ sudo xl sched-credit
Cpupool Pool-0: <span class="nv">tslice</span><span class="o">=</span>30ms <span class="nv">ratelimit</span><span class="o">=</span>1000us
Name ID Weight Cap
Domain-0 <span class="m">0</span> <span class="m">512</span> <span class="m">0</span>
Wheezy <span class="m">3</span> <span class="m">256</span> <span class="m">0</span>
</pre></div>
<p><a href="https://www.kernel.org/">Download</a> the latest kernel version you would like to use and extract
the contents of the archive. You can try pulling your configuration via
“<em>make oldconfig</em>“, so your old settings are migrated and only new or
changed options are presented to you to select. Then to make sure
everything is ok, run “<em>make menuconfig</em>” or “<em>make xconfig</em>” to
determine if the feature/module setting are appropriate for you. I left
everything alone with the exception of enabling the Xen features as
described below. <em>make oldconfig</em> is clever, it can do its job between
different versions of kernel although just issuing a “<em>make menuconfig</em>”
is probably also fine.</p>
<div class="highlight"><pre><span></span>$ <span class="nb">cd</span> linux-3.9.2
$ make oldconfig
scripts/kconfig/conf --oldconfig Kconfig
<span class="c1">#</span>
<span class="c1"># configuration written to .config</span>
<span class="c1">#</span>
</pre></div>
<p>Alternatively just use the defaults and add the required Xen features:</p>
<div class="highlight"><pre><span></span>$ <span class="nb">cd</span> linux-3.9.2
$ make menuconfig
</pre></div>
<p>Location:<br>
-> Processor type and features<br>
-> Paravirtualized guest support<br>
Select all features.</p>
<p><img alt="makemenu3" src="https://www.rsreese.com/assets/makemenu3.png"></p>
<p><img alt="makemenu2" src="https://www.rsreese.com/assets/makemenu2.png"></p>
<p><img alt="makemenu4" src="https://www.rsreese.com/assets/makemenu4.png"></p>
<p>Location:<br>
-> Device Drivers<br>
-> Block devices<br>
Select the two features “Xen virtual block device support” and “Xen
block-device backend driver”</p>
<p><img alt="makemenu5" src="https://www.rsreese.com/assets/makemenu5.png"></p>
<p><img alt="makemenu6" src="https://www.rsreese.com/assets/makemenu6.png"></p>
<p><img alt="makemenu7" src="https://www.rsreese.com/assets/makemenu7.png"></p>
<p>Location:<br>
-> Device Drivers<br>
-> Xen driver support<br>
Select all features.</p>
<p><img alt="makemenu5" src="https://www.rsreese.com/assets/makemenu5.png"></p>
<p><img alt="makemenu8" src="https://www.rsreese.com/assets/makemenu8.png"></p>
<p><img alt="makemenu9" src="https://www.rsreese.com/assets/makemenu9.png"></p>
<p>Location:<br>
-> Device Drivers<br>
-> Network device support<br>
Select the two features “Xen network device frontend driver” and “Xen
backend network device”</p>
<p><img alt="makemenu5" src="https://www.rsreese.com/assets/makemenu5.png"></p>
<p><img alt="makemenu10" src="https://www.rsreese.com/assets/makemenu10.png"></p>
<p><img alt="makemenu11" src="https://www.rsreese.com/assets/makemenu11.png"></p>
<p>Lastly, you can search using “/” when at the root menu to see what you
have enabled:</p>
<p><img alt="makemenu1" src="https://www.rsreese.com/assets/makemenu1.png"></p>
<p>Which will provide you a list of features that have been selected but it
may be easier to grep through the .config as shown in the next command.</p>
<p><img alt="makemenu12" src="https://www.rsreese.com/assets/makemenu12.png"></p>
<p>You can use “<em>grep</em>” to ensure you should have similar values for your
Xen settings after running menu config.</p>
<div class="highlight"><pre><span></span>$ grep XEN .config
<span class="nv">CONFIG_XEN</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_DOM0</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_PRIVILEGED_GUEST</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_PVHVM</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_MAX_DOMAIN_MEMORY</span><span class="o">=</span><span class="m">500</span>
<span class="nv">CONFIG_XEN_SAVE_RESTORE</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_DEBUG_FS</span><span class="o">=</span>y
<span class="nv">CONFIG_PCI_XEN</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_PCIDEV_FRONTEND</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_BLKDEV_FRONTEND</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_BLKDEV_BACKEND</span><span class="o">=</span>y
<span class="nv">CONFIG_NETXEN_NIC</span><span class="o">=</span>m
<span class="nv">CONFIG_XEN_NETDEV_FRONTEND</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_NETDEV_BACKEND</span><span class="o">=</span>y
<span class="nv">CONFIG_INPUT_XEN_KBDDEV_FRONTEND</span><span class="o">=</span>y
<span class="nv">CONFIG_HVC_XEN</span><span class="o">=</span>y
<span class="nv">CONFIG_HVC_XEN_FRONTEND</span><span class="o">=</span>y
<span class="c1"># CONFIG_XEN_WDT is not set</span>
<span class="nv">CONFIG_XEN_FBDEV_FRONTEND</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_BALLOON</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_BALLOON_MEMORY_HOTPLUG</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_SCRUB_PAGES</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_DEV_EVTCHN</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_BACKEND</span><span class="o">=</span>y
<span class="nv">CONFIG_XENFS</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_COMPAT_XENFS</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_SYS_HYPERVISOR</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_XENBUS_FRONTEND</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_GNTDEV</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_GRANT_DEV_ALLOC</span><span class="o">=</span>y
<span class="nv">CONFIG_SWIOTLB_XEN</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_PCIDEV_BACKEND</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_PRIVCMD</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_ACPI_PROCESSOR</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_MCE_LOG</span><span class="o">=</span>y
<span class="nv">CONFIG_XEN_HAVE_PVMMU</span><span class="o">=</span>y
</pre></div>
<p>If all of the Xen features are enabled, move on to compiling.</p>
<div class="highlight"><pre><span></span>$ make bzImage
$ make modules
$ sudo make modules_install
</pre></div>
<p>Copy the images to the appropriate locations.</p>
<div class="highlight"><pre><span></span>$ sudo cp -a arch/x86/boot/bzImage /boot/vmlinuz-3.9.2
$ sudo cp -a System.map /boot/System.map-3.9.2
$ sudo cp -a .config /boot/config-3.9.2
$ sudo depmod -a
$ sudo mkinitrd /boot/initrd.img-3.9.2 <span class="m">3</span>.9.2
</pre></div>
<p>Add a grub entry to /etc/grub.conf, make sure it is the first entry but
leave an existing distribution kernel entry to fall back to if there are problems:</p>
<div class="highlight"><pre><span></span>title Xen 4.2.1 / Kernel 3.9.2
root (hd0,0)
kernel /xen.gz
module /vmlinuz-3.9.2
module /initrd.img-3.9.2
</pre></div>
<p>Reboot the system and you should be able to run the following command to
verify that your efforts have paid off.</p>
<div class="highlight"><pre><span></span>$ sudo xl list
Name ID Mem VCPUs State Time<span class="o">(</span>s<span class="o">)</span>
Domain-0 <span class="m">0</span> <span class="m">2048</span> <span class="m">1</span> r----- <span class="m">941</span>.4
</pre></div>
<p>Now you can move on to setting up a guest as described in <a href="http://www.rsreese.com/creating-debian-guests-on-xen-using-parition-based-filesystem/" title="Creating Debian guests on Xen using parition based filesystem">Creating
Debian guests on Xen using parition based filesystem</a>.</p>
<p>If you are unable to reboot using your new kernel, revert back to a
distro kernel and double check that you have done everything as
described. If something is not clear or could be improved upon, let me
know by leaving a comment below.</p>Passive DNS collection and analysis using YaF and Mediator2013-05-20T11:12:00-04:002013-05-20T11:12:00-04:00Stephen Reesetag:www.rsreese.com,2013-05-20:/passive-dns-collection-and-analysis-using-yaf-and-mediator/Passive DNS is a useful tool for any analysts teams toolbox, I have noted several public sensors here but they only see data (queries and responses) that transverse their sensors. I have been working on setting up passive DNS using Yet another Flowmeter (YaF) and Mediator (YaF to MySQL) to …<p>Passive <span class="caps">DNS</span> is a useful tool for any analysts teams toolbox, I have noted several public sensors <a href="http://www.rsreese.com/online-information-security-analysis-tools-and-resources/">here</a> but they only see data (queries and responses) that transverse their sensors. I have been working on setting up passive <span class="caps">DNS</span> using Yet another Flowmeter (YaF) and Mediator (YaF to MySQL) to fill the gap where third-party sensors may not be providing the coverage I would like. Passive <span class="caps">DNS</span> can provide tremendous insight and analytics upon <span class="caps">DNS</span> queries that users and/or malware may beperforming. A few items of interest:</p>
<ul>
<li>Hostnames that have a large number of <span class="caps">IP</span> addresses associated with them in a short time period and they have only been visited by very few hosts host on the network.</li>
<li>Tertiary name usage associated with a specific domain?</li>
<li>When was the domain first resolved on the network and further, how often is it being resolved and by whom?</li>
<li>A recently accessed/registered domain with short time to live (TTLs) often associated with new <span class="caps">IP</span> addresses may indicate
malicious activity, or a <span class="caps">CDN</span>.</li>
<li>Queries for TLDs that you typically do not interact with may be worth looking into.</li>
<li>Users using non-approved <span class="caps">DNS</span> servers</li>
</ul>
<p>Passive <span class="caps">DNS</span> may be also helpful in tracking infections using Fast-fluxwhich make blocking the C2 difficult as the attackers will create algorithms to rotate the <span class="caps">IP</span> addresses and even the hostnames in the case of double-flux. (<a href="http://www.cs.ucsb.edu/~kemm/courses/cs177/torpig.pdf">TorPig</a>) The list goes on but in a nutshell, I wanted to be able to perform this activity without having to rely on having all of the <span class="caps">DNS</span> server logs in a centralized location, especially since users may reconfigure their <span class="caps">DNS</span> settings to use non-approved servers, e.g. <span class="caps">BYOD</span>.</p>
<p>This entry demonstrates how to build and setup <a href="http://tools.netsa.cert.org/yaf/yaf.html">YaF</a> and <a href="https://tools.netsa.cert.org/confluence/pages/viewpage.action?pageId=15958035">Mediator</a> both of which are available from the <span class="caps">CERT</span> <a href="http://www.cert.org/netsa/">NetSA</a> <a href="http://tools.netsa.cert.org/">site</a> and should be considered complementary to the documentation the NetSA team have already provided for each of the respective tools. This setup was tested on CentOS 6.4 but most Linux distributions should work fine.</p>
<ol>
<li>Have site reconfigure interfaces on all hosts. eth0 should be management interface and eth1 should be the tap <span class="caps">OR</span> whatever makes sense, this need to happen every time the host comes up, i.e.</li>
</ol>
<div class="highlight"><pre><span></span>sudo ifconfig eth1 up promisc
</pre></div>
<ol>
<li>Ensure development libraries/dependencies are installed. Some may require enabling the optional software channel</li>
</ol>
<div class="highlight"><pre><span></span>sudo yum install glib2-devel lzo gcc-c++ libpcap-devel pcre-devel
</pre></div>
<ol>
<li>Install <a href="http://tools.netsa.cert.org/fixbuf/">libfixbuf</a></li>
</ol>
<div class="highlight"><pre><span></span>cd libfixbuf-1.3.0
./configure
make
sudo make install
</pre></div>
<ol>
<li>Install YaF</li>
</ol>
<div class="highlight"><pre><span></span>cd yaf-2.3.3
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
./configure --with-libpcap --enable-applabel --enable-plugins
make
sudo make install
</pre></div>
<ol>
<li>Edit ld</li>
</ol>
<div class="highlight"><pre><span></span>sudo echo "/usr/local/lib" >> /etc/ld.so.conf
sudo /sbin/ldconfig
sudo /sbin/ldconfig -v | grep libzmq # should rebuild the cache including zmq too.
</pre></div>
<p><span class="caps">OR</span></p>
<div class="highlight"><pre><span></span>export PATH=$PATH:/usr/local/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
</pre></div>
<ol>
<li>Configure cmake</li>
</ol>
<div class="highlight"><pre><span></span>cd cmake-2.8.10.2
./configure
gmake
</pre></div>
<ol>
<li>Optionally, configure YaF to File output for testing purposes.</li>
</ol>
<div class="highlight"><pre><span></span>export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
cd yaf_file_mediator-1.1.0/
./configure
../cmake-2.8.10.2/bin/cmake .
make
</pre></div>
<ol>
<li>Configure YaF to MySQL</li>
</ol>
<div class="highlight"><pre><span></span>export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
cd yaf_silk_mysql_mediator-1.4.0
../cmake-2.8.10.2/bin/cmake .
./configure --with-mysql
make
</pre></div>
<p>Next, populate create a database and respective tables:</p>
<div class="highlight"><pre><span></span>./yafMySQL -o localhost -n username -p password -d eflows
</pre></div>
<ol>
<li>Setup YaF to start capturing. Here we are only capture <span class="caps">DNS</span> traffic and rotating the files written to disk after 5 minutes. Originally set to 10 minutes but yaf_silk_mysql_mediator would segmentation fault because MySQL would close the connection before all of the data would insert. We have a continuous method that works a little better which we should a little later. We lock the file so that another process cannot take the file that is currently being written to.</li>
</ol>
<div class="highlight"><pre><span></span>sudo /usr/local/bin/yaf --live pcap --in eth1 --out /data/ipfix/ --rotate 600 --filter="port 53" --applabel --applabel-rules=/usr/local/etc/yafApplabelRules.conf --max-payload=1000 --plugin-name=/usr/local/lib/yaf/dpacketplugin.la --plugin-opts="53" --lock --become-user=nobody --become-group=nobody &
</pre></div>
<ol>
<li>Testing the output of a YaF</li>
</ol>
<div class="highlight"><pre><span></span>yaf_file_mediator-1.1.0/yaf_file_mediator --input /data/ipfix/filename.yaf --output test.txt
</pre></div>
<p>After a few minutes, you should be able to parse the filename.yaf that was first written (in this case 5 minutes). The contents of test.txt should be similar to the following:</p>
<div class="highlight"><pre><span></span>-------------------------------
Template ID is 45841
Application Label: 53
Source IP: 192.168.0.5
Destination IP: 8.8.8.8
Source Port: 53855
Dest Port: 53
Flow Attributes: 1
Rev Flow Attributes: 0
flowStartTime: 2013-04-24 23:53:43
flowEndTime: 2013-04-24 23:58:02
flowEndReason: 1
Protocol: 17
Octet Total Count: 120
Rev Octet count: 244
Packet Total Count: 2
Rev Packet Total Count: 2
DNS ID: 32852 Type: 28 RR Section: 0 TTL: 0 Query: www.google.com.
DNS ID: 32852 Type: 28 RR Section: 1 TTL: 204 RRName: www.google.com. AAAA: 2607:f8b0:400c:0c04::0069
-------------------------------
Template ID is 45841
Application Label: 53
Source IP: 192.168.0.5
Destination IP: 8.8.8.8
Source Port: 50845
Dest Port: 53
flowStartTime: 2013-04-24 23:58:02
flowEndTime: 2013-04-24 23:58:02
flowEndReason: 1
Protocol: 17
Octet Total Count: 60
Rev Octet count: 156
Packet Total Count: 1
Rev Packet Total Count: 1
DNS ID: 21141 Type: 1 RR Section: 0 TTL: 0 Query: www.google.com.
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.103
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.99
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.105
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.104
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.106
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.147
</pre></div>
<ol>
<li>After you have confirmed that your YaF entries contain records, adda little automation. This will scoop up the files in the directory where the YaF files are being written, place them in the MySQL <span class="caps">DBMS</span> and delete the file. Note, if you start seeing “Segmentation Fault” then MySQL maybe closing the connection before all of the records from the YaF file could be written to the <span class="caps">DBMS</span>. You can try modifying MySQL parameters or reduce the the size of YaF files being written to disk in order to try mitigating this symptom if it occurs in your environment.</li>
</ol>
<div class="highlight"><pre><span></span>for i in $( ls /data/ipfix/*.yaf ); do /home/user/silk-installs/yaf_silk_mysql_mediator-1.4.0/yaf_silk_mysql_mediator --in-file $i --mysql-host localhost --name username --pass password --database eflows && sudo rm $i; done
</pre></div>
<p>Here is our first query, lets see who has recently made requests for www.google.com.</p>
<div class="highlight"><pre><span></span>mysql> SELECT rrname,rrval,srcip4,dstip4,flowStartMilliseconds FROM dns d, flows f WHERE f.id = d.id AND rrname LIKE "www.google.com." GROUP by rrval ORDER BY f.id DESC LIMIT 50;
+-----------------+---------------------------+------------+-----------+-----------------------+
| rrname | rrval | srcip4 | dstip4 | flowStartMilliseconds |
+-----------------+---------------------------+------------+-----------+-----------------------+
| www.google.com. | 2001:4860:4001:0802::1012 | 3232235525 | 134744072 | 2013-05-03 17:47:24 |
| www.google.com. | 2001:4860:4001:0801::1014 | 3232235525 | 134744072 | 2013-05-03 15:35:32 |
| www.google.com. | 2001:4860:4001:0802::1014 | 3232235525 | 134744072 | 2013-05-03 11:28:42 |
| www.google.com. | 2001:4860:4001:0801::1010 | 3232235525 | 134744072 | 2013-05-02 16:48:31 |
| www.google.com. | 2001:4860:4001:0802::1011 | 3232235525 | 134744072 | 2013-05-02 13:33:57 |
| www.google.com. | 2001:4860:4001:0803::1010 | 3232235525 | 134744072 | 2013-05-02 12:01:56 |
| www.google.com. | 2607:f8b0:4004:0801::1012 | 3232235525 | 134744072 | 2013-05-01 21:36:55 |
| www.google.com. | 2001:4860:4001:0802::1010 | 3232235525 | 134744072 | 2013-05-01 12:44:52 |
| www.google.com. | 74.125.239.80 | 3232235525 | 134744072 | 2013-05-01 10:45:04 |
| www.google.com. | 74.125.239.83 | 3232235525 | 134744072 | 2013-05-01 10:45:04 |
| www.google.com. | 74.125.239.82 | 3232235525 | 134744072 | 2013-05-01 10:45:04 |
| www.google.com. | 74.125.239.81 | 3232235525 | 134744072 | 2013-05-01 10:45:04 |
| www.google.com. | 74.125.239.84 | 3232235525 | 134744072 | 2013-05-01 10:45:04 |
| www.google.com. | 2607:f8b0:4004:0802::1010 | 3232235525 | 134744072 | 2013-04-29 19:54:00 |
| www.google.com. | 2607:f8b0:4005:0802::1010 | 3232235525 | 134744072 | 2013-04-28 15:52:00 |
| www.google.com. | 2607:f8b0:4004:0803::1013 | 3232235525 | 134744072 | 2013-04-28 15:05:53 |
| www.google.com. | 2607:f8b0:4005:0802::1011 | 3232235525 | 134744072 | 2013-04-27 14:45:35 |
| www.google.com. | 2607:f8b0:4004:0801::1013 | 3232235525 | 134744072 | 2013-04-26 18:53:45 |
| www.google.com. | 2607:f8b0:4005:0802::1012 | 3232235525 | 134744072 | 2013-04-26 13:55:51 |
| www.google.com. | 2607:f8b0:4005:0802::1013 | 3232235525 | 134744072 | 2013-04-26 12:35:18 |
| www.google.com. | 74.125.239.145 | 3232235525 | 134744072 | 2013-04-26 12:03:10 |
| www.google.com. | 74.125.239.148 | 3232235525 | 134744072 | 2013-04-26 12:03:10 |
| www.google.com. | 74.125.239.146 | 3232235525 | 134744072 | 2013-04-26 12:03:10 |
| www.google.com. | 74.125.239.147 | 3232235525 | 134744072 | 2013-04-26 12:03:10 |
| www.google.com. | 74.125.239.144 | 3232235525 | 134744072 | 2013-04-26 12:03:10 |
| www.google.com. | 2607:f8b0:4005:0802::1014 | 3232235525 | 134744072 | 2013-04-26 11:31:59 |
| www.google.com. | 74.125.228.112 | 3232235525 | 134744072 | 2013-04-25 16:25:39 |
| www.google.com. | 74.125.228.114 | 3232235525 | 134744072 | 2013-04-25 16:25:39 |
| www.google.com. | 74.125.228.113 | 3232235525 | 134744072 | 2013-04-25 16:25:39 |
| www.google.com. | 74.125.228.115 | 3232235525 | 134744072 | 2013-04-25 16:25:39 |
| www.google.com. | 74.125.228.116 | 3232235525 | 134744072 | 2013-04-25 16:25:39 |
| www.google.com. | 2607:f8b0:4004:0802::1012 | 3232235525 | 134744072 | 2013-04-25 11:29:45 |
| www.google.com. | 2607:f8b0:4004:0803::1014 | 3232235525 | 134744072 | 2013-04-24 20:33:42 |
| www.google.com. | 2607:f8b0:400e:0c04::006a | 3232235525 | 134744072 | 2013-04-24 18:04:19 |
| www.google.com. | 2607:f8b0:400e:0c02::006a | 3232235525 | 134744072 | 2013-04-24 15:26:22 |
| www.google.com. | 74.125.228.20 | 3232235525 | 134744072 | 2013-04-24 12:05:43 |
| www.google.com. | 74.125.228.16 | 3232235525 | 134744072 | 2013-04-24 12:05:43 |
| www.google.com. | 74.125.228.18 | 3232235525 | 134744072 | 2013-04-24 12:05:43 |
| www.google.com. | 74.125.228.19 | 3232235525 | 134744072 | 2013-04-24 12:05:43 |
| www.google.com. | 74.125.228.17 | 3232235525 | 134744072 | 2013-04-24 12:05:43 |
| www.google.com. | 2607:f8b0:4004:0801::1014 | 3232235525 | 134744072 | 2013-04-23 20:43:26 |
| www.google.com. | 74.125.228.50 | 3232235525 | 134744072 | 2013-04-23 20:38:43 |
| www.google.com. | 74.125.228.51 | 3232235525 | 134744072 | 2013-04-23 20:38:43 |
| www.google.com. | 74.125.228.52 | 3232235525 | 134744072 | 2013-04-23 20:38:43 |
| www.google.com. | 74.125.228.48 | 3232235525 | 134744072 | 2013-04-23 20:38:43 |
| www.google.com. | 74.125.228.49 | 3232235525 | 134744072 | 2013-04-23 20:38:43 |
| www.google.com. | 2607:f8b0:4004:0801::1011 | 3232235525 | 134744072 | 2013-04-23 18:38:52 |
| www.google.com. | 2607:f8b0:400e:0c01::0067 | 3232235525 | 134744072 | 2013-04-23 15:57:45 |
| www.google.com. | 2607:f8b0:4004:0801::1010 | 3232235525 | 134744072 | 2013-04-23 15:07:59 |
| www.google.com. | 2607:f8b0:400e:0c01::0069 | 3232235525 | 134744072 | 2013-04-23 12:30:28 |
+-----------------+---------------------------+------------+-----------+-----------------------+
</pre></div>
<p>Here is a similar query but we want to see any tertiary youtube.com domains and sort by the lookup returned.</p>
<div class="highlight"><pre><span></span>mysql> SELECT qr,type,auth,nx,ttl,rrname,rrval from dns WHERE rrname LIKE "%.youtube.com." GROUP BY rrval LIMIT 50;
+------+------+------+------+------+--------------------------------+----------------+
| qr | type | auth | nx | ttl | rrname | rrval |
+------+------+------+------+------+--------------------------------+----------------+
| 0 | 1 | 0 | 0 | 0 | www.youtube.com. | |
| 1 | 1 | 0 | 0 | 300 | v17.lscache2.c.youtube.com. | 12.216.80.12 |
| 1 | 1 | 0 | 0 | 1800 | r2.sn-5uu-vgqe.c.youtube.com. | 12.216.80.13 |
| 1 | 1 | 0 | 0 | 1800 | r3.sn-5uu-vgqe.c.youtube.com. | 12.216.80.14 |
| 1 | 1 | 0 | 0 | 1800 | r4.att-ord1.c.youtube.com. | 12.216.80.15 |
| 1 | 1 | 0 | 0 | 1800 | r6.sn-5uu-vgqe.c.youtube.com. | 12.216.80.17 |
| 1 | 1 | 0 | 0 | 1714 | r8.sn-5uu-vgqe.c.youtube.com. | 12.216.80.19 |
| 1 | 1 | 0 | 0 | 1741 | r1.sn-5uu-vgql.c.youtube.com. | 12.216.80.44 |
| 1 | 1 | 0 | 0 | 1800 | r2.sn-5uu-vgql.c.youtube.com. | 12.216.80.45 |
| 1 | 1 | 0 | 0 | 1800 | r3.sn-5uu-vgql.c.youtube.com. | 12.216.80.46 |
| 1 | 1 | 0 | 0 | 1279 | r4.sn-5uu-vgql.c.youtube.com. | 12.216.80.47 |
| 1 | 1 | 0 | 0 | 1800 | r6.sn-5uu-vgql.c.youtube.com. | 12.216.80.49 |
| 1 | 1 | 0 | 0 | 1800 | r7.sn-5uu-vgql.c.youtube.com. | 12.216.80.50 |
| 1 | 1 | 0 | 0 | 1739 | r8.sn-5uu-vgql.c.youtube.com. | 12.216.80.51 |
| 1 | 1 | 0 | 0 | 1800 | r12.sn-hp576nes.c.youtube.com. | 173.194.17.17 |
| 1 | 1 | 0 | 0 | 1800 | r20.sn-hp576nes.c.youtube.com. | 173.194.17.25 |
| 1 | 1 | 0 | 0 | 1800 | r6.sn-q4f7dnel.c.youtube.com. | 173.194.24.11 |
| 1 | 1 | 0 | 0 | 1800 | r1.dfw06s08.c.youtube.com. | 173.194.24.134 |
| 1 | 1 | 0 | 0 | 1800 | r15.sn-q4f7dn7r.c.youtube.com. | 173.194.24.148 |
| 1 | 1 | 0 | 0 | 1800 | r18.sn-hp576n7d.c.youtube.com. | 173.194.29.119 |
| 1 | 1 | 0 | 0 | 1800 | r9.sn-hp576n7z.c.youtube.com. | 173.194.29.46 |
| 1 | 1 | 0 | 0 | 1800 | r5.sn-ab5e6ner.c.youtube.com. | 173.194.31.10 |
| 1 | 1 | 0 | 0 | 1800 | r1.sn-ab5e6nle.c.youtube.com. | 173.194.31.102 |
| 1 | 1 | 0 | 0 | 640 | r2.sn-ab5e6nle.c.youtube.com. | 173.194.31.103 |
| 1 | 1 | 0 | 0 | 1800 | r3.sn-ab5e6nle.c.youtube.com. | 173.194.31.104 |
| 1 | 1 | 0 | 0 | 1800 | r4.sn-ab5e6nle.c.youtube.com. | 173.194.31.105 |
| 1 | 1 | 0 | 0 | 1800 | r5.sn-ab5e6nle.c.youtube.com. | 173.194.31.106 |
| 1 | 1 | 0 | 0 | 1800 | r6.sn-ab5e6nle.c.youtube.com. | 173.194.31.107 |
| 1 | 1 | 0 | 0 | 1800 | r7.sn-ab5e6nle.c.youtube.com. | 173.194.31.108 |
| 1 | 1 | 0 | 0 | 1800 | r8.sn-ab5e6nle.c.youtube.com. | 173.194.31.109 |
| 1 | 1 | 0 | 0 | 705 | r6.sn-ab5e6ner.c.youtube.com. | 173.194.31.11 |
| 1 | 1 | 0 | 0 | 1800 | r9.sn-ab5e6nle.c.youtube.com. | 173.194.31.110 |
| 1 | 1 | 0 | 0 | 1800 | r10.sn-ab5e6nle.c.youtube.com. | 173.194.31.111 |
| 1 | 1 | 0 | 0 | 292 | r11.sn-ab5e6nle.c.youtube.com. | 173.194.31.112 |
| 1 | 1 | 0 | 0 | 1800 | r12.sn-ab5e6nle.c.youtube.com. | 173.194.31.113 |
| 1 | 1 | 0 | 0 | 178 | r13.sn-ab5e6nle.c.youtube.com. | 173.194.31.114 |
| 1 | 1 | 0 | 0 | 1800 | r14.sn-ab5e6nle.c.youtube.com. | 173.194.31.115 |
| 1 | 1 | 0 | 0 | 1800 | r15.sn-ab5e6nle.c.youtube.com. | 173.194.31.116 |
| 1 | 1 | 0 | 0 | 1800 | r16.sn-ab5e6nle.c.youtube.com. | 173.194.31.117 |
| 1 | 1 | 0 | 0 | 1800 | r17.sn-ab5e6nle.c.youtube.com. | 173.194.31.118 |
| 1 | 1 | 0 | 0 | 1800 | r18.sn-ab5e6nle.c.youtube.com. | 173.194.31.119 |
| 1 | 1 | 0 | 0 | 1653 | r7.sn-ab5e6ner.c.youtube.com. | 173.194.31.12 |
| 1 | 1 | 0 | 0 | 1800 | r19.sn-ab5e6nle.c.youtube.com. | 173.194.31.120 |
| 1 | 1 | 0 | 0 | 1800 | r20.sn-ab5e6nle.c.youtube.com. | 173.194.31.121 |
| 1 | 1 | 0 | 0 | 1800 | r8.sn-ab5e6ner.c.youtube.com. | 173.194.31.13 |
| 1 | 1 | 0 | 0 | 81 | r1.sn-ab5e6nll.c.youtube.com. | 173.194.31.134 |
| 1 | 1 | 0 | 0 | 1800 | r2.sn-ab5e6nll.c.youtube.com. | 173.194.31.135 |
| 1 | 1 | 0 | 0 | 1800 | r3.sn-ab5e6nll.c.youtube.com. | 173.194.31.136 |
| 1 | 1 | 0 | 0 | 1800 | r4.sn-ab5e6nll.c.youtube.com. | 173.194.31.137 |
| 1 | 1 | 0 | 0 | 1800 | r5.lga15s22.c.youtube.com. | 173.194.31.138 |
+------+------+------+------+------+--------------------------------+----------------+
50 rows in set (22.17 sec)
</pre></div>
<p>An alternative method is to write YaF records directly to mediator, and further the MySQL <span class="caps">DBMS</span> rather then writing files to disk although youcan still do this with the appropriate toggles. Here is example usage to start the processes:</p>
<div class="highlight"><pre><span></span>$ ./silk-installs/yaf_silk_mysql_mediator-1.4.0/yaf_silk_mysql_mediator --in-host<span class="o">=</span><span class="m">127</span>.0.0.1 --in-port<span class="o">=</span><span class="m">18000</span> --mysql-host<span class="o">=</span>localhost --name<span class="o">=</span>username --pass password --database eflows
$ sudo /usr/local/bin/yaf --live pcap --in eth1 --out <span class="m">127</span>.0.0.1 --ipfix-port<span class="o">=</span><span class="m">18000</span> --ipfix tcp --log<span class="o">=</span>/var/log/yaf.log --filter<span class="o">=</span><span class="s2">"port 53"</span> --applabel --applabel-rules<span class="o">=</span>/usr/local/etc/yafApplabelRules.conf --max-payload<span class="o">=</span><span class="m">1000</span> --plugin-name<span class="o">=</span>/usr/local/lib/yaf/dpacketplugin.la --plugin-opts<span class="o">=</span><span class="s2">"53"</span> <span class="p">&</span>
</pre></div>
<p>Ensure YaF and mediator are connected:</p>
<div class="highlight"><pre><span></span>$ sudo netstat -tupan<span class="p">|</span>grep yaf
tcp <span class="m">0</span> <span class="m">0</span> <span class="m">127</span>.0.0.1:18000 <span class="m">0</span>.0.0.0:* LISTEN <span class="m">6497</span>/yaf_silk_mysql
tcp <span class="m">0</span> <span class="m">0</span> <span class="m">127</span>.0.0.1:47417 <span class="m">127</span>.0.0.1:18000 ESTABLISHED <span class="m">6513</span>/yaf
tcp <span class="m">0</span> <span class="m">0</span> <span class="m">127</span>.0.0.1:18000 <span class="m">127</span>.0.0.1:47417 ESTABLISHED <span class="m">6497</span>/yaf_silk_mysql
</pre></div>
<p>You may use the following MySQL query to see when the table was last updated to ensure records are being inserted on a regular basis:</p>
<div class="highlight"><pre><span></span>mysql> SHOW TABLE STATUS in eflows;
</pre></div>
<p>After a few minutes of collection, query a domain that has been recently resolved and you should see it in the <span class="caps">DBMS</span>.</p>
<div class="highlight"><pre><span></span>mysql> SELECT rrname,rrval from dns WHERE rrname LIKE "%rsreese.com." GROUP BY rrval LIMIT 10;
+--------------+--------------------------------+
| rrname | rrval |
+--------------+--------------------------------+
| rsreese.com. | |
| rsreese.com. | 2600:3c02::f03c:91ff:fe96:f7bd |
| rsreese.com. | 74.207.234.79 |
| rsreese.com. | ns1.linode.com. |
| rsreese.com. | ns2.linode.com. |
| rsreese.com. | ns3.linode.com. |
| rsreese.com. | ns4.linode.com. |
| rsreese.com. | ns5.linode.com. |
+--------------+--------------------------------+
8 rows in set (18.26 sec)
</pre></div>
<p>There are a number of different fields available for query so I leave it to you to come up with whatever is most useful for you. Further, think of how you could write a shiny front-end for analysts to use rather then having to use the MySQL command line interface. Hope you found this useful and leave a comment if you did or have any questions. </p>Running Moloch2013-03-16T18:19:00-04:002013-03-16T18:19:00-04:00Stephen Reesetag:www.rsreese.com,2013-03-16:/running-moloch/This is an overview of installing and running Moloch on a single host. After seeing the 2013 ShmooCon presentation, I have been looking forward to giving the tool a test-drive. Per the documentation, “Moloch is a open source large scale IPv4 full PCAP capturing, indexing and database system”. It is …<p>This is an overview of installing and running <a href="https://github.com/aol/moloch#what-is-moloch">Moloch</a> on a single host. After seeing the 2013 ShmooCon <a href="http://www.shmoocon.org/speakers#moloch">presentation</a>, I have been looking forward to giving the tool a test-drive. Per the documentation, “Moloch is a open source large scale IPv4 full <span class="caps">PCAP</span> capturing, indexing and database system”. It is fast and has a pretty nice interface to boot. Although it does not contain the same feature-set as some commercial over the shelf (<span class="caps">COTS</span>) products, I see Moloch fitting into a similar space where <span class="caps">COTS</span> products such might sit. When analysts are made aware of anomaly-based alerts from signature/misuse based intrusion detection systems (<span class="caps">IDS</span>), e.g. Snort, or anomalous activity from network flow, e.g. SiLK, the analyst can obtain packet capture (<span class="caps">PCAP</span>) for further investigation. The existing commercial tool suites are expensive <span class="caps">PCAP</span> indexing tools if that is all they are being used for, especially if you are locked into their storage mechanism. A budget conscious security operation center (<span class="caps">SOC</span>) can setup Moloch for a fraction of the maintenance cost of commercial offerings and instead use the funds for additional hardware (longer retention), maintenance, and even some Moloch development contribution. </p>
<p>Although the developers have provided a script to get Moloch going, I had a few hiccups so I figured I would document them in the event they help someone else out. I used a CentOS release 6.4 (Final) x86_64 base bare-metal install. I imagine you could run it in a virtual environment for testing purposes. After you get the operating system (<span class="caps">OS</span>) installed and patched, pull down the latest Oracle Java for your distribution. Untar the package and create a symbolic in a directory that Moloch will be able to find. </p>
<div class="highlight"><pre><span></span>$ sudo cp -R jre1.7.0_17/ /usr/bin/
$ sudo ln -s /usr/bin/jre1.7.0_17/bin/java /usr/bin/java
</pre></div>
<p>Next, pull down the latest moloch build. I just grabbed the <span class="caps">ZIP</span> but it is hosted on GitHub. You might want to take a look at the install script to see if everything is ideal for you. Run the easy installer which should pull down the prerequisites needed, build and install. </p>
<div class="highlight"><pre><span></span>$ <span class="nb">cd</span> moloch-master/
$ sudo ./easybutton-singlehost.sh
</pre></div>
<p>If everything went smoothly, the script will try starting the three Moloch components being elasticsearch, capture, and viewer. The latter process did not start and this was probably for the better as I required me to take a closer look at what the install script was doing and the default configuration files (<em>config.ini</em> and <em>elaseticsearch.yml</em>). The configuration files are located in: </p>
<div class="highlight"><pre><span></span># ls -l /data/moloch/etc/
total 4680
-rw-r--r--. 1 root root 6766 Mar 14 17:21 config.ini
-rw-r--r--. 1 root root 6551 Mar 13 22:30 config.ini.template
-rw-r--r--. 1 root root 12545 Mar 14 22:54 elasticsearch.yml
-rw-r--r--. 1 root root 3360134 Mar 6 15:10 GeoIPASNum.dat
-rw-r--r--. 1 root root 1358092 Mar 5 21:48 GeoIP.dat
-rw-r--r--. 1 root root 1249 Mar 13 22:31 moloch.crt
-rw-r--r--. 1 root root 1029 Mar 13 22:31 moloch.csr
-rw-r--r--. 1 root root 1704 Mar 13 22:31 moloch.key
-rw-r--r--. 1 root root 10875 Mar 13 22:31 openssl.cnf
-rw-r--r--. 1 root root 10909 Mar 13 22:30 openssl.cnf.template
</pre></div>
<p>First, I had to sort out what was preventing the viewer from starting so I took a look at the viewer.log.</p>
<div class="highlight"><pre><span></span>Mar 13 23:13:04 http.c:245 moloch_http_connect(): Connecting 0x7f6e0d19b010
Mar 13 23:13:04 http.c:276 moloch_http_connect(): 0x7f6e0d19b010: Error: Error connecting: Address family not supported by protocol
Couldn't connect to elastic search at 'localhost:9200'
</pre></div>
<p>Log files are located in:</p>
<div class="highlight"><pre><span></span># ls -l /data/moloch/logs/
total 6047776
-rw-r--r--. 1 root root 6180585472 Mar 15 23:44 capture.log
-rw-r--r--. 1 root root 12062720 Mar 14 17:22 capture.log.old
-rw-r--r--. 1 root root 0 Mar 13 22:31 Moloch_index_indexing_slowlog.log
-rw-r--r--. 1 root root 0 Mar 13 22:31 Moloch_index_search_slowlog.log
-rw-r--r--. 1 root root 163 Mar 15 20:00 Moloch.log
-rw-r--r--. 1 root root 2943 Mar 13 23:27 Moloch.log.2013-03-13
-rw-r--r--. 1 root root 35410 Mar 14 23:34 Moloch.log.2013-03-14
-rw-r--r--. 1 root root 208487 Mar 15 23:06 viewer.log
-rw-r--r--. 1 root root 1668 Mar 15 09:06 viewer.log.old
</pre></div>
<p>I had to change the directive in the config.ini from localhost to 127.0.0.1, otherwise the viewer would not connect to the elasticsearch instance in CentOS. Probably due to the initial IPv6 look-up, just a guess. Also added a Berkley packet filter (<span class="caps">BPF</span>) to prevent the capture and indexing of internal-to-internal traffic. </p>
<div class="highlight"><pre><span></span>elasticsearch=127.0.0.1:9200
bpf=not src net (10.0.0.0/8) and dst net (10.0.0.0/8)
</pre></div>
<p>While I was adjusting the configuration, I decided to adjust the elasticsearch memory usage from what I originally specified in the installer script. You might want to take a look at their <a href="https://github.com/aol/moloch#hardware-requirements">hardware requirements</a> but I was able to run with a less powerful node: </p>
<p><em>$ sudo vim /data/moloch/bin/run_es.sh</em></p>
<div class="highlight"><pre><span></span>ES_HEAP_SIZE=2G bin/elasticsearch -Des.config=<span class="cp">${</span><span class="n">TDIR</span><span class="cp">}</span>/etc/elasticsearch.yml
</pre></div>
<p>The viewer would now start (the capture and viewer process were already running but had gracefully killed them). Here are the commands to start each process based on the default installation criteria.</p>
<div class="highlight"><pre><span></span>$ sudo nohup /data/moloch/bin/run_es.sh
$ sudo nohup /data/moloch/bin/run_capture.sh <span class="p">&</span>
$ sudo nohup /data/moloch/bin/run_viewer.sh <span class="p">&</span>
</pre></div>
<p>Sessions page screen-shot after capturing some traffic, not including session listing:</p>
<p><a href="https://www.rsreese.com/assets/moloch-graph.png"><img alt="Moloch Graph" src="https://www.rsreese.com/assets/moloch-graph-thumb.jpg"></a></p>
<p>Stats page screen-shot:</p>
<p><a href="https://www.rsreese.com/assets/moloch-stats.png"><img alt="moloch-stats-thumb" src="https://www.rsreese.com/assets/moloch-stats-thumb.jpg"></a></p>
<p>I noticed the mention of two plugins to keep tabs on the elasticsearch memory usage and to maintain session data. This is pretty important as I determined if you remove <span class="caps">PCAP</span> and the session data remained, think metadata, users that attempted to drill-down on the aforementioned session data for the missing <span class="caps">PCAP</span> would cause the viewer process to die. In my case, I setup Putty to tunnel my connection to the locally listening plug-in interfaces and delete the offending session data: </p>
<p><img alt="moloch-putty" src="https://www.rsreese.com/assets/moloch-putty.png"></p>
<p>ElasticSearch maintenance screenshot located at http://127.0.0.1:9200/_plugin/head/ after tunneling via Putty. I was able to drop the session via this interface.</p>
<p><a href="https://www.rsreese.com/assets/moloch-head.png"><img alt="moloch-head-thumb" src="https://www.rsreese.com/assets/moloch-head-thumb.jpg"></a></p>
<p>Node statistics screen-shot accessed at
http://127.0.0.1:9200/_plugin/bigdesk/ after correctly configuring
Putty. Note that we want to keep an eye on the heap memory to ensure it
does not approach the maximum specified value. There are many more
statistics not shown in this screen-shot.</p>
<p><a href="https://www.rsreese.com/assets/moloch-bigdesk.png"><img alt="moloch-bigdesk-thumb" src="https://www.rsreese.com/assets/moloch-bigdesk-thumb.jpg"></a></p>
<p>Here is a Youtube <a href="http://www.youtube.com/watch?v=BWxrXJz_Ay0">video</a> featuring Moloch in actions. As usual, if you have trouble installing or running Moloch, please leave a comment below, and do not forget to check out the Moloch <a href="https://github.com/aol/moloch/wiki/FAQ"><span class="caps">FAQ</span></a>.</p>Increment IP packet timestamp2013-03-13T02:48:00-04:002013-03-13T02:48:00-04:00Stephen Reesetag:www.rsreese.com,2013-03-13:/increment-ip-packet-timestamp/I recently had a need to specify and increment the IP timestamp values of packets in a PCAP. In this example, the starting second value is specified and we increment the microsecond value. This requires the use of Scapy. If you have any questions or recommendations for improvement, please leave …<p>I recently had a need to specify and increment the <span class="caps">IP</span> timestamp values of packets in a <code>PCAP</code>. In this example, the starting second value is specified and we increment the microsecond value. This requires the use of <a href="http://www.secdev.org/projects/scapy/">Scapy</a>. If you have any questions or recommendations for improvement, please leave a comment below.</p>
<table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/python</span>
<span class="c1"># Script to parse a PCAP and modify timestamps</span>
<span class="c1"># Requires Scapy</span>
<span class="c1"># 0.1 - 03012012</span>
<span class="c1"># Stephen Reese</span>
<span class="kn">from</span> <span class="nn">scapy.all</span> <span class="kn">import</span> <span class="o">*</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="c1"># Get input and output files from command line</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o"><</span> <span class="mi">2</span><span class="p">:</span>
<span class="k">print</span> <span class="s2">"Usage: rewritetimestamp.py inputpcapfile"</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># Assign variable names for input and output files</span>
<span class="n">infile</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">process_packets</span><span class="p">():</span>
<span class="n">pkts</span> <span class="o">=</span> <span class="n">rdpcap</span><span class="p">(</span><span class="n">infile</span><span class="p">)</span>
<span class="n">cooked</span><span class="o">=</span><span class="p">[]</span>
<span class="n">timestamp</span> <span class="o">=</span> <span class="mf">1234567890.000000</span>
<span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">pkts</span><span class="p">:</span>
<span class="n">p</span><span class="o">.</span><span class="n">time</span> <span class="o">=</span> <span class="n">timestamp</span>
<span class="n">timestamp</span> <span class="o">+=</span> <span class="mf">0.000001</span>
<span class="n">pmod</span><span class="o">=</span><span class="n">p</span>
<span class="n">p</span><span class="o">.</span><span class="n">time</span>
<span class="n">cooked</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">pmod</span><span class="p">)</span>
<span class="n">wrpcap</span><span class="p">(</span><span class="s2">"out.pcap"</span><span class="p">,</span> <span class="n">cooked</span><span class="p">)</span>
<span class="n">process_packets</span><span class="p">()</span>
</pre></div>
</td></tr></table>Running SnortAD2013-01-10T03:00:00-05:002013-01-10T03:00:00-05:00Stephen Reesetag:www.rsreese.com,2013-01-10:/running-snortad/I recently fired up a Snort Anomaly Detection instance provided by the SnortAD project and wanted to share my experience for those who might be interested in trying it on your network. SnortAD is the third generation anomaly detection preprocessor for Snort and is a little different than its predecessors …<p>I recently fired up a Snort Anomaly Detection instance provided by the
<a href="http://anomalydetection.info/">SnortAD</a> project and wanted to share my experience for those who
might be interested in trying it on your network. SnortAD is the third
generation anomaly detection preprocessor for Snort and is a little
different than its predecessors but don’t take my word for it, check out
their <a href="http://anomalydetection.info/">site</a>.</p>
<p>First you need to create a log file based on your network, the log file
will contain a profile of your network traffics characteristics.
Although a log file has been provided with the SnortAD virtual machine
(<span class="caps">VM</span>) that contains null entries it will not do you much good aside from
alerting on everything. In order to characterize your network, you will
need to create a log file with enough data to be statistically relevant.
For the impatient, you can create a day or two worth of data and
duplicate the data. Duplicating the data will have adverse effects
though. Think about a university in which a majority of classes occur on
Monday and Wednesday. If you only create a profile for Monday and
duplicate it for the rest of the week, you can quickly understand how
your results might be skewed.</p>
<p>To get going, use the snort.conf included on SnortAD <span class="caps">VM</span> and begin
creating a log file but remember to backup or remove the original log
file in the event you need it for reference. Also, always backup your
configuration files before making changes for good measure.</p>
<p>Configure the snort.conf file to log. Something like the following
should work fine:</p>
<div class="highlight"><pre><span></span>preprocessor AnomalyDetection: LogPath /var/log/snort log time 60
</pre></div>
<p></code></p>
<p>Next, run Snort to generate log data. As mentioned, you should create
enough data to make it statistically relevant. The evaluator script
expects three weeks. As an alternate, you might be able to use tcpreplay
to replay existing <span class="caps">PCAP</span> if you have enough data.</p>
<div class="highlight"><pre><span></span>$ sudo /usr/local/bin/snort -c /etc/snort.conf -i eth0
</pre></div>
<p>You should start seeing messages to stdout that look like the following:</p>
<div class="highlight"><pre><span></span>Loged transfer between 06-01-13 15:33:52 - 06-01-13 15:34:52
Loged transfer between 06-01-13 15:34:52 - 06-01-13 15:35:52
</pre></div>
<p>Now you should have a log with a number of entries saved in
/var/log/snort. The profile generation script is next run. In this
example we specify a week rather than opt for the three week default but
again, <span class="caps">YMMV</span> and you made need to adjust these values. Also, make sure
you check the help of the profile generator as there are other
algorithms, five to be specific: Moving average (default), Naive method,
Autoregressive time series model, Holt-Winters model, and <span class="caps">HW</span> model with
Brutlag’s confidence band.</p>
<div class="highlight"><pre><span></span>/usr/local/src/profilegenerator/ad_profilegenerator.r -m AVG --avg 'WEEKLY,1' -l Log_Data.txt -p profile.txt -e evaluator.txt -P pattern.txt
</pre></div>
<p>The previous command creates the profile.txt file which is a <span class="caps">CSV</span> file,
i.e. you could respectively name it profile.csv. The <span class="caps">CSV</span> file will be
used by your updated snort.conf file. In order to enable anomaly
detection, we need to download or create a few Snort configuration files:</p>
<div class="highlight"><pre><span></span>$ ls -l /etc/snort
total <span class="m">4200</span>
-rw-r--r--. <span class="m">1</span> root root <span class="m">3621</span> Jan <span class="m">5</span> <span class="m">15</span>:35 classification.config
-rw-r--r--. <span class="m">1</span> root root <span class="m">29596</span> Jan <span class="m">5</span> <span class="m">15</span>:35 gen-msg.map
-rw-r--r--. <span class="m">1</span> root root <span class="m">7897</span> Jan <span class="m">5</span> <span class="m">15</span>:35 preprocessor.rules
-rw-r--r--. <span class="m">1</span> root root <span class="m">1484013</span> Jan <span class="m">5</span> <span class="m">15</span>:35 profile.csv
-rw-r--r--. <span class="m">1</span> root root <span class="m">746</span> Jan <span class="m">5</span> <span class="m">15</span>:35 reference.config
-rw-r--r--. <span class="m">1</span> root root <span class="m">2696705</span> Jan <span class="m">5</span> <span class="m">15</span>:35 sid-msg.map
-rw-r--r--. <span class="m">1</span> root root <span class="m">255</span> Jan <span class="m">5</span> <span class="m">15</span>:35 snort.conf
-rw-r--r--. <span class="m">1</span> root root <span class="m">2556</span> Jan <span class="m">5</span> <span class="m">15</span>:35 threshold.conf
-rw-r--r--. <span class="m">1</span> root root <span class="m">53841</span> Jan <span class="m">5</span> <span class="m">15</span>:35 unicode.map
</pre></div>
<p>I found it simplest to pull down the latest Snort signature as they have
the additional required files that are not included in the provide
SnortAD build. You can pull down the needed preprocessor.rules from one
of the authors <a href="https://bitbucket.org/AnomalyDetection/preprocessor/src/2aaea35a15b0a3dcb7f627cc428e4a136420c9d3/preproc_rules/preprocessor.rules?at=default">bitbucket</a>. The snort.conf was populated with the
following contents:</p>
<div class="highlight"><pre><span></span>include classification.config
include reference.config
include preprocessor.rules
preprocessor AnomalyDetection: ProfilePath /etc/snort/profile.csv LogPath /var/log/snort alert log time 60
</pre></div>
<p>If you have everything in the /etc/snort directory, you should be able
to run Snort and see alerts when anomalies are detected:</p>
<div class="highlight"><pre><span></span>$ sudo /usr/local/bin/snort -c /etc/snort/snort.conf -i eth0
</pre></div>
<p>Here are some sample alerts from some early testing. It will probably
take some tuning to begin seeing useful alerts:</p>
<div class="highlight"><pre><span></span><span class="k">[**] [1000100:1000101:1] AD_UNUSUALLY_HIGH_TCP_TRAFFIC [**]</span>
<span class="k">[Classification: Potentially Bad Traffic] [Priority: 2]</span>
<span class="na">01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8</span>
<span class="na">ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF</span>
<span class="na">Type:8 Code:0 ID:30537 Seq:1 ECHO</span>
<span class="k">[**] [1000100:1000107:1] AD_HIGH_LAN_TCP_TRAFFIC [**]</span>
<span class="k">[Classification: Potentially Bad Traffic] [Priority: 2]</span>
<span class="na">01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8</span>
<span class="na">ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF</span>
<span class="na">Type:8 Code:0 ID:30537 Seq:1 ECHO</span>
<span class="k">[**] [1000100:1000108:1] AD_UNUSUALLY_LOW_UDP_TRAFFIC [**]</span>
<span class="k">[Classification: Potentially Bad Traffic] [Priority: 2]</span>
<span class="na">01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8</span>
<span class="na">ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF</span>
<span class="na">Type:8 Code:0 ID:30537 Seq:1 ECHO</span>
<span class="k">[**] [1000100:1000114:1] AD_LOW_LAN_UDP_TRAFFIC [**]</span>
<span class="k">[Classification: Potentially Bad Traffic] [Priority: 2]</span>
<span class="na">01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8</span>
<span class="na">ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF</span>
<span class="na">Type:8 Code:0 ID:30537 Seq:1 ECHO</span>
<span class="k">[**] [1000100:1000134:1] AD_LOW_ARP_REQUEST_NUMBER [**]</span>
<span class="k">[Classification: Potentially Bad Traffic] [Priority: 2]</span>
<span class="na">01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8</span>
<span class="na">ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF</span>
<span class="na">Type:8 Code:0 ID:30537 Seq:1 ECHO</span>
<span class="k">[**] [1000100:1000138:1] AD_LOW_NOT_TCP_IP_TRAFFIC [**]</span>
<span class="k">[Classification: Potentially Bad Traffic] [Priority: 2]</span>
<span class="na">01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8</span>
<span class="na">ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF</span>
<span class="na">Type:8 Code:0 ID:30537 Seq:1 ECHO</span>
<span class="k">[**] [1000100:1000140:1] AD_LOW_OVERALL_PACKET_NUMBER [**]</span>
<span class="k">[Classification: Potentially Bad Traffic] [Priority: 2]</span>
<span class="na">01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8</span>
<span class="na">ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF</span>
<span class="na">Type:8 Code:0 ID:30537 Seq:1 ECHO</span>
</pre></div>
<p>If you have any questions, leave a comment and/or check out the authors
<a href="https://bitbucket.org/AnomalyDetection/preprocessor/raw/2aaea35a15b0a3dcb7f627cc428e4a136420c9d3/ReadMe.txt">Readme.txt</a> for some additional usage insight.</p>Mailing Lists2012-11-10T04:27:00-05:002012-11-10T04:27:00-05:00Stephen Reesetag:www.rsreese.com,2012-11-10:/mailing-lists/Here are a few technology and information security related mailing-lists that I subscribe to in no particular order. Leave a comment if you think I missed one. asterisk-users.lists.digium.com beginners.perl.org snort-users.lists.sourceforge.net nessus.list.nessus.org pauldotcom.mail.pauldotcom.com samurai-devel.lists.sourceforge.net …<p>Here are a few technology and information security related mailing-lists
that I subscribe to in no particular order. Leave a comment if you think
I missed one.</p>
<p><em>asterisk-users.lists.digium.com</em><br>
<em>beginners.perl.org</em><br>
<em>snort-users.lists.sourceforge.net</em><br>
<em>nessus.list.nessus.org</em><br>
<em>pauldotcom.mail.pauldotcom.com</em><br>
<em>samurai-devel.lists.sourceforge.net</em><br>
<em>ptk-forensics-mail.lists.sourceforge.net</em><br>
<em>gcfa.lists.sans.org</em><br>
<em>framework-hackers.spool.metasploit.com</em><br>
<em>framework.spool.metasploit.com</em><br>
<em>secureideas-base-user.lists.sourceforge.net</em><br>
<em>python-list.python.org</em><br>
<em>nexpose-users.lists.rapid7.com</em><br>
<em>winquisitor-beta.googlegroups.com</em><br>
<em>securitybsides.googlegroups.com</em><br>
<em>datarecoverycertification.googlegroups.com</em><br>
<em>full-disclosure.lists.grok.org.uk</em><br>
<em>scap_interest.ietf.org</em><br>
<em>cipp.news.infracritical.com</em><br>
<em>scadasec.news.infracritical.com</em><br>
<em>debian-security-announce.lists.debian.org</em><br>
<em>bugtraq.list-id.securityfocus.com</em><br>
<em>ietf.ietf.org</em><br>
<em>dfir.lists.sans.org</em><br>
<em>webappsec.list-id.securityfocus.com</em><br>
<em>sleuthkit-users.lists.sourceforge.net</em><br>
<em>vol-users.volatilesystems.com</em><br>
<em>emerging-sigs.emergingthreats.net</em></p>Podcasts2012-09-10T04:29:00-04:002012-09-10T04:29:00-04:00Stephen Reesetag:www.rsreese.com,2012-09-10:/podcasts/Here is a list of information technology and security podcasts. Some are technical, others are higher level so YMMV. A source of information to keep me up to date on what is going on in the information technology realm. If you think of something I have missed, leave a commment …<p>Here is a list of information technology and security podcasts. Some are technical, others are higher level so <span class="caps">YMMV</span>. A source of information to keep me up to date on what is going on in the information technology realm. If you think of something I have missed, leave a commment. Some of these may be explicit so please use discretion and they are in no particular order.</p>
<p><a href="http://dataclonelabs.com/security_talkworkshop/datasecurity.xml" title="http://www.thecyberjungle.com">The CyberJungle</a><br>
<a href="http://www.cbsnews.com/common/includes/podcast/podcast_larry_magid_1.rss" title="http://www.cbsradionewsfeed.com/rss.php?id=112">Tech Talk</a><br>
<a href="http://feeds2.feedburner.com/Threatpost-DigitalUnderground" title="http://threatpost.com/en_us/feeds/blog/hearsay/digitalunderground.xml">The Digital Underground Podcast</a><br>
<a href="http://feeds.feedburner.com/TheLinuxActionShow" title="http://www.jupiterbroadcasting.com">The Linux Action Show! <span class="caps">MP3</span></a><br>
<a href="http://feeds.packetpushers.net/PacketPushersPodcast" title="http://packetpushers.net">Packet Pushers Podcast</a><br>
<a href="http://www.cigital.com/silverbullet/feed/" title="http://www.cigital.com/silverbullet">The Silver Bullet Security Podcast</a><br>
<a href="http://www.social-engineer.org/category/podcast/" title="http://socialengineer.podbean.com">Social-Engineer.Org PodCast</a><br>
<a href="http://sfspodcast.libsyn.com/rss" title="http://www.southernfriedsecurity.com">The Southern Fried Security Podcast</a><br>
<a href="https://isc.sans.edu/dailypodcast.xml" title="http://isc.sans.edu/">Internet Storm Center Threat Update</a><br>
<a href="http://risky.biz/feeds/rb2" title="http://risky.biz/feeds/rb2"><span class="caps">RB2</span></a><br>
<a href="http://www.npr.org/rss/podcast.php?id=1019&uid=n1qe4e85742c986fdb81d2d38ffa0d5d53" title="http://www.npr.org/templates/topics/topic.php?topicId= 1019"><span class="caps">NPR</span> Topics: Technology Podcast</a><br>
<a href="http://securityweekly.com/podcast/psw.xml" title="http://www.securityweekly.com/">Security Weekly</a><br>
<a href="http://feeds.sophos.com/en/rss2_0-sophos-podcasts.xml" title="http://feeds.sophos.com/en/rss2_0-sophos-podcasts.xml">Sophos Podcasts</a><br>
<a href="http://risky.biz/feeds/risky-business" title="http://risky.biz/feeds/risky-business">Risky Business</a><br>
<a href="http://www.eurotrashsecurity.eu/episodes/eurotrash.xml" title="http://www.eurotrashsecurity.eu">Eurotrash Security Podcast: Security with funny accents</a><br>
<a href="http://securabit.libsyn.com/rss" title="http://www.securabit.com">SecuraBit</a><br>
<a href="http://www.securitycatalyst.com/feed/" title="http://www.securitycatalyst.com">The Security Catalyst</a><br>
<a href="http://www.cert.org/podcast/exec_podcast.rss" title="http://www.cert.org/podcast"><span class="caps">CERT</span> Podcast Series: Security for Business Leaders</a><br>
<a href="http://feeds2.feedburner.com/myharddrivedied" title="http://podnutz.com/mhdd/feed">My Hard Drive Died - w/Scott Moulton</a><br>
<a href="https://www.owasp.org/download/jmanico/podcast.xml" title="https://www.owasp.org/index.php/OWASP_Podcast"><span class="caps">OWASP</span> Security Podcast</a><br>
<a href="http://crypto-gram.libsyn.com/rss" title="http://crypto-gram.libsyn.com">Crypto-Gram Security Podcast</a><br>
<a href="http://www.cisco.com/assets/cdc_content_elements/rss/security_podcast/security_tac_pcast.xml" title="http://www.cisco.com/en/US/solutions/ns170/tac/ security_tac_podcasts.html">Cisco <span class="caps">TAC</span> Security Podcast Series</a><br>
<a href="http://www.2600.com/oth-broadband.xml" title="http://www.2600.com/offthehook/">Off The Hook: high-bitrate <span class="caps">MP3</span> feed</a><br>
<a href="http://infosecplacepodcast.com/?feed=podcast" title="http://infosecplacepodcast.com">An Information Security Place Podcast</a><br>
<a href="http://feeds.feedburner.com/townsendsecurity" title="http://www.townsendsecurity.com">Security Insider - Podcast Edition</a><br>
<a href="http://feeds.wsjonline.com/wsj/podcast_wall_street_journal_tech_news_briefing?format=xml" title="http://online.wsj.com/page/audio.html">Wall Street Journal Tech News Briefing</a><br>
<a href="http://feeds.feedburner.com/The404?format=xml" title="http://www.cnet.com/8300-13952_1-81.html">The 404 (<span class="caps">MP3</span>)</a><br>
<a href="http://downloads.bbc.co.uk/podcasts/worldservice/digitalp/rss.xml" title="http://www.bbc.co.uk/click">Click</a><br>
<a href="http://ashimmy.podomatic.com/rss2.xml" title="http://ashimmy.podomatic.com">Security.Exe powered by The <span class="caps">CISO</span> Group with Alan Shimel</a><br>
<a href="http://leoville.tv/podcasts/floss.xml" title="http://twit.tv"><span class="caps">FLOSS</span> Weekly</a><br>
<a href="http://blog.stackoverflow.com/?feed=podcast" title="http://blog.stackoverflow.com">The Stack Exchange Podcast</a><br>
<a href="http://podcast.wh1t3rabbit.net/rss" title="http://hp.com/go/white-rabbit">Down the Security Rabbithole</a><br>
<a href="http://auditcasts.com/screencasts/feed.rss" title="http://auditcasts.com/">AuditCasts with David Hoelzer</a><br>
<a href="http://feeds.feedburner.com/CeriasSecuritySeminarPodcast" title="http://www.cerias.purdue.edu/security_seminar"><span class="caps">CERIAS</span> Security Seminar Podcast</a></p>Decoding XOR payload using first few bytes as key2012-07-24T04:07:00-04:002012-07-24T04:07:00-04:00Stephen Reesetag:www.rsreese.com,2012-07-24:/decoding-xor-payload-using-first-few-bytes-as-key/I recently came across the need to decode an exclusive or (XOR) payload. In my case, the key to de-obfuscating the traffic was the first three bytes of each packets payload. While it is trivial to decode each payload, it was not reasonable for a large number of packets. For …<p>I recently came across the need to decode an exclusive or (<span class="caps">XOR</span>) payload.
In my case, the key to de-obfuscating the traffic was the first three
bytes of each packets payload. While it is trivial to decode each
payload, it was not reasonable for a large number of packets.</p>
<p>For testing purposes, create a packet:</p>
<div class="highlight"><pre><span></span>$ scapy
Welcome to Scapy <span class="o">(</span><span class="m">2</span>.1.0<span class="o">)</span>
>>> <span class="nv">p</span> <span class="o">=</span> <span class="o">(</span>IP<span class="o">(</span><span class="nv">ttl</span><span class="o">=</span><span class="m">10</span><span class="o">)</span>/TCP<span class="o">(</span><span class="nv">sport</span><span class="o">=</span><span class="m">1024</span>,dport<span class="o">=</span><span class="m">443</span>,flags<span class="o">=</span><span class="s2">"S"</span><span class="o">)</span>/<span class="s2">" WHATSTHESECRET0000ABCD0000ABCD0000ABCD"</span><span class="o">)</span>
>>> wrpcap<span class="o">(</span><span class="s2">"p.pcap"</span>, p<span class="o">)</span>
>>> quit<span class="o">()</span>
</pre></div>
<p>Should see something similar to this:</p>
<div class="highlight"><pre><span></span><span class="mi">04</span><span class="o">:</span><span class="mi">29</span><span class="o">:</span><span class="mf">31.255470</span> <span class="n">IP</span> <span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="o">.</span><span class="mi">1024</span> <span class="o">></span> <span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="o">.</span><span class="mi">443</span><span class="o">:</span> <span class="n">Flags</span> <span class="o">[</span><span class="n">S</span><span class="o">],</span> <span class="n">seq</span> <span class="mi">0</span><span class="o">:</span><span class="mi">41</span><span class="o">,</span> <span class="n">win</span> <span class="mi">8192</span><span class="o">,</span> <span class="n">length</span> <span class="mi">41</span>
<span class="mh">0x0000</span><span class="o">:</span> <span class="mi">4500</span> <span class="mi">0051</span> <span class="mi">0001</span> <span class="mi">0000</span> <span class="mi">0</span><span class="n">a06</span> <span class="n">b2a4</span> <span class="mi">7</span><span class="n">f00</span> <span class="mi">0001</span> <span class="n">E</span><span class="o">..</span><span class="n">Q</span><span class="o">............</span>
<span class="mh">0x0010</span><span class="o">:</span> <span class="mi">7</span><span class="n">f00</span> <span class="mi">0001</span> <span class="mi">0400</span> <span class="mi">01</span><span class="n">bb</span> <span class="mi">0000</span> <span class="mi">0000</span> <span class="mi">0000</span> <span class="mi">0000</span> <span class="o">................</span>
<span class="mh">0x0020</span><span class="o">:</span> <span class="mi">5002</span> <span class="mi">2000</span> <span class="mi">751</span><span class="n">d</span> <span class="mi">0000</span> <span class="mi">2020</span> <span class="mi">2057</span> <span class="mi">4841</span> <span class="mi">5453</span> <span class="n">P</span><span class="o">...</span><span class="n">u</span><span class="o">......</span><span class="n">WHATS</span>
<span class="mh">0x0030</span><span class="o">:</span> <span class="mi">5448</span> <span class="mi">4553</span> <span class="mi">4543</span> <span class="mi">5245</span> <span class="mi">5430</span> <span class="mi">3030</span> <span class="mi">3041</span> <span class="mi">4243</span> <span class="n">THESECRET0000ABC</span>
<span class="mh">0x0040</span><span class="o">:</span> <span class="mi">4430</span> <span class="mi">3030</span> <span class="mi">3041</span> <span class="mi">4243</span> <span class="mi">4430</span> <span class="mi">3030</span> <span class="mi">3041</span> <span class="mi">4243</span> <span class="n">D0000ABCD0000ABC</span>
<span class="mh">0x0050</span><span class="o">:</span> <span class="mi">44</span> <span class="n">D</span>
</pre></div>
<p><img alt="Screen Shot" src="https://www.rsreese.com/assets/Screen-Shot-2012-07-23-at-10.37.49-PM.png"></p>
<p>Next, the payload is <span class="caps">XOR</span> using the first three bytes of the payload
for the entire payload. If you note the first tcpdump, the three bytes
of the payload were left empty, here I am placing the key that will be
used to <span class="caps">XOR</span> the rest of the payload within the first three bytes of the payload.</p>
<p><img alt="Screen Shot" src="https://www.rsreese.com/assets/Screen-Shot-2012-07-23-at-10.39.30-PM.png"></p>
<p>The payload has been obfuscated using the key ‘the’.</p>
<p>Next we can use the <a href="https://code.google.com/p/reese/source/browse/trunk/decodexorpayload.py">script</a> below or <a href="https://code.google.com/p/reese/source/browse/trunk/decodexorpayload.py">here</a> to decode all of
the packets. The script is not intelligent enough to know which need to
be de-obfuscated so it is best to probably filter these into a new <span class="caps">PCAP</span>.
Secondly, the script requires <a href="http://www.secdev.org/projects/scapy/">Scapy</a> to be installed.</p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/python</span>
<span class="c1"># Script to parse a PCAP and XOR data based on a byte offset</span>
<span class="c1"># Requires Scapy</span>
<span class="c1"># 0.1 - 07172012</span>
<span class="c1"># Default is two bytes, change at line 35</span>
<span class="c1"># Stephen Reese and Chris Gragsone</span>
<span class="c1">#</span>
<span class="c1"># todo: add two more args, offset length and static offset option</span>
<span class="kn">from</span> <span class="nn">scapy.all</span> <span class="kn">import</span> <span class="o">*</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="c1"># Get input and output files from command line</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o"><</span> <span class="mi">2</span><span class="p">:</span>
<span class="k">print</span> <span class="s2">"Usage: decodexorpayload.py [input pcap file]"</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># Assign variable names for input and output files</span>
<span class="n">infile</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">many_byte_xor</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span>
<span class="n">buf</span> <span class="o">=</span> <span class="nb">bytearray</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span>
<span class="n">key</span> <span class="o">=</span> <span class="nb">bytearray</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
<span class="n">key_len</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">bufbyte</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">buf</span><span class="p">):</span>
<span class="n">buf</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">bufbyte</span> <span class="o">^</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span> <span class="o">%</span> <span class="n">key_len</span><span class="p">]</span>
<span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">process_packets</span><span class="p">():</span>
<span class="n">pkts</span> <span class="o">=</span> <span class="n">rdpcap</span><span class="p">(</span><span class="n">infile</span><span class="p">)</span>
<span class="n">cooked</span><span class="o">=</span><span class="p">[]</span>
<span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">pkts</span><span class="p">:</span>
<span class="c1"># You may have to adjust the payload depth here:</span>
<span class="c1"># i.e. p.payload.payload.payload</span>
<span class="n">pkt_payload</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">payload</span><span class="o">.</span><span class="n">payload</span><span class="p">)</span>
<span class="n">pkt_offset</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">payload</span><span class="o">.</span><span class="n">payload</span><span class="p">)[:</span><span class="mi">3</span><span class="p">]</span>
<span class="k">if</span> <span class="n">pkt_payload</span> <span class="ow">and</span> <span class="n">pkt_offset</span><span class="p">:</span>
<span class="n">pmod</span><span class="o">=</span><span class="n">p</span>
<span class="c1"># You may have to adjust the payload depth here:</span>
<span class="n">p</span><span class="o">.</span><span class="n">payload</span><span class="o">.</span><span class="n">payload</span><span class="o">=</span><span class="n">many_byte_xor</span><span class="p">(</span><span class="n">pkt_payload</span><span class="p">,</span> <span class="n">pkt_offset</span><span class="p">)</span>
<span class="n">cooked</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">pmod</span><span class="p">)</span>
<span class="n">wrpcap</span><span class="p">(</span><span class="s2">"dump.pcap"</span><span class="p">,</span> <span class="n">cooked</span><span class="p">)</span>
<span class="n">process_packets</span><span class="p">()</span>
</pre></div>
<p>After script completion, viewing the packet does indeed show the
de-obfuscated packet:</p>
<div class="highlight"><pre><span></span>reading from file dump.pcap, link-type RAW (Raw IP)
04:24:44.415262 IP 127.0.0.1.1024 > 127.0.0.1.443: Flags [S], seq 0:41, win 8192, length 41
0x0000: 4500 0051 0001 0000 0a06 b2a4 7f00 0001 E..Q............
0x0010: 7f00 0001 0400 01bb 0000 0000 0000 0000 ................
0x0020: 5002 2000 751d 0000 0000 0057 4841 5453 P...u......WHATS
0x0030: 5448 4553 4543 5245 5430 3030 3041 4243 THESECRET0000ABC
0x0040: 4430 3030 3041 4243 4430 3030 3041 4243 D0000ABCD0000ABC
0x0050: 44 D
</pre></div>
<p>There are a number of features that could be added and of course the
code can probably be improved upon. Have some ideas? Leave a comment below.</p>World IPv6 Day2012-06-07T01:29:00-04:002012-06-07T01:29:00-04:00Stephen Reesetag:www.rsreese.com,2012-06-07:/world-ipv6-day/World IPv6 Day on June 8th 2012 is rapidly approaching. It is an exciting and scary reality. For my personal assets, there was a small investment on my part to get everything up to par. My internet provider Comcast is dual-stack ready which is nice because I experienced some serious …<p><img src="/assets/IPv6-wordmark-256-trans.png" style="float:right; padding:10px;" /><a href="http://www.worldipv6day.org/">World IPv6 Day</a> on June 8th 2012 is rapidly approaching. It is an exciting and scary reality. For my personal assets, there was a small investment on my part to get everything up to par. My internet provider Comcast is dual-stack ready which is nice because I experienced some serious latency from time to time when using a tunnel-broker (note that other factors probably contributed). You can see more information about the Comcast IPv6 trial and preparation <a href="http://www.comcast6.net/">here</a>. First, I had to invest in a new cable-modem as my old Motorola <span class="caps">SB1000</span> was not up to the task. Comcast has created a hardware compatibility <a href="http://mydeviceinfo.comcast.net/">list</a>. From the list I decided to go with the Motorola <span class="caps">SB6121</span> as I have had pretty good success with their modems in the past. Secondly you need a device that is capable of filtering and distributing addresses to your internal devices. I am not going into details here, but a Cisco <span class="caps">ASA5500</span> or a home-brew Linux device usually will work quite nicely. The most important part to read into is that you are also filtering v6 <span class="caps">IP</span> traffic along with the v4 so you do not have evil-doers sneaker-netting into your network. Your network devices will not hide behind network address translation (<span class="caps">NAT</span>). Lastly, keep the images, firmware, or distributions patched and monitor your traffic from time to time. Kind of like a cavity, you usually do not know you have one until it is too late.</p>
<p>My blog has also moved to a dual-stack (<a href="http://www.linode.com/?r=6579d0b21f581ea769a6ca4af46de0dad6f88df8">Linode</a> awesome service and support) from a tunnel-broker! This was really straightforward to implement as Linode provides some great documentation in their <a href="https://library.linode.com/networking/ipv6">library</a>. As with any setup, you need to filter unwanted traffic from entering/exiting your node(s), Iptables makes quick work of this. In this scenario, I am going with a deny-by-default posture and log everything that is dropped. This is by no means definitive but just a place to get started. </p>
<div class="highlight"><pre><span></span>*filter
# Drop everything
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
# Allow the loopback
-A INPUT -i lo -j ACCEPT
-A INPUT -d ::1/128 ! -i lo -j REJECT --reject-with icmp6-port-unreachable
# All returning connections
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# Let the web server respond
-A INPUT -p tcp --sport 1024:65535 --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -p tcp --sport 1024:65535 --dport 443 -m state --state NEW -j ACCEPT
# All SSH session but limit attempt, also see fail2ban
-A INPUT -p tcp --sport 1024:65535 --dport 22 --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -m limit --limit 1/min --limit-burst 3 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
# Allow ICMP but need to restrict based on type
-A INPUT -p ipv6-icmp -j ACCEPT
# Drop everything else and log it
-A INPUT -m limit --limit 3/min -j LOG --log-prefix "ipv6 input denied: " --log-level 7
# Respective outbound rules
-A OUTPUT -p ipv6-icmp -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -m limit --limit 3/min -j LOG --log-prefix "ipv6 output denied: " --log-level 7
COMMIT
</pre></div>How-to setup an Upside-Down-Ternet2012-02-11T03:07:00-05:002012-02-11T03:07:00-05:00Stephen Reesetag:www.rsreese.com,2012-02-11:/how-to-setup-an-upside-down-ternet/In an effort to replicate the amusing idea of a transparent proxy that manipulates traffic in a fun way found here and made even better with some great scripts that you can pull down from here. A Debian box was stood up with two network cards; one connects to the …<p>In an effort to replicate the amusing idea of a transparent proxy that
manipulates traffic in a fun way found <a href="http://www.ex-parrot.com/pete/upside-down-ternet.html">here</a> and made even better
with some great scripts that you can pull down from <a href="https://code.google.com/p/g0tmi1k/source/browse/trunk#trunk%2FsquidScripts">here</a>. A Debian
box was stood up with two network cards; one connects to the internal
<span class="caps">LAN</span> and the other connected to an access-point which your guests connect
to. I chose to post this how-to as the initial idea did not provide a
complete reference on how to setup the needed components.</p>
<p>First, we are using an access-point we take care of the <span class="caps">DHCP</span> and <span class="caps">DNS</span>
duties but the access-point or another host could perform these duties
if they support said services. I choose to install the following <span class="caps">DHCP</span> service:</p>
<div class="highlight"><pre><span></span>$ sudo apt-get install isc-dhcp-server
</pre></div>
<p>The following configuration provides the scope for the clients. We only
define a scope for the client side which will use a 192.168.0.0 network
for the example purposes.</p>
<div class="highlight"><pre><span></span>$ grep ^<span class="o">[</span>^#<span class="o">]</span> /etc/dhcp/dhcpd.conf
ddns-update-style none<span class="p">;</span>
default-lease-time <span class="m">600</span><span class="p">;</span>
max-lease-time <span class="m">7200</span><span class="p">;</span>
log-facility local7<span class="p">;</span>
subnet <span class="m">192</span>.168.0.0 netmask <span class="m">255</span>.255.255.0 <span class="o">{</span>
range <span class="m">192</span>.168.0.100 <span class="m">192</span>.168.0.200<span class="p">;</span>
option domain-name-servers <span class="m">192</span>.168.0.1<span class="p">;</span>
option domain-name <span class="s2">"kittenwar.com"</span><span class="p">;</span>
option routers <span class="m">192</span>.168.0.1<span class="p">;</span>
option broadcast-address <span class="m">192</span>.168.0.255<span class="p">;</span>
default-lease-time <span class="m">600</span><span class="p">;</span>
max-lease-time <span class="m">7200</span><span class="p">;</span>
<span class="o">}</span>
</pre></div>
<p>Secondly, the guests are going to need some resolution, rather than have
their queries pass through the network, lets setup a simple resolver for
them using <span class="caps">BIND</span>:</p>
<div class="highlight"><pre><span></span>$ sudo apt-get install bind9
</pre></div>
<p>Setup some forwarders and the interface we want to listen on, for
example sake, the same subnet servicing the clients:</p>
<div class="highlight"><pre><span></span>$ grep ^<span class="o">[</span>^#<span class="o">]</span> /etc/bind/named.conf.options
options <span class="o">{</span>
directory <span class="s2">"/var/cache/bind"</span><span class="p">;</span>
version <span class="s2">"tbd"</span><span class="p">;</span>
forwarders <span class="o">{</span> <span class="m">8</span>.8.8.8<span class="p">;</span> <span class="m">8</span>.8.4.4<span class="p">;</span> <span class="o">}</span><span class="p">;</span>
auth-nxdomain no<span class="p">;</span> <span class="c1"># conform to RFC1035</span>
listen-on-v6 <span class="o">{</span> none<span class="p">;</span> <span class="o">}</span><span class="p">;</span>
listen-on <span class="o">{</span> <span class="m">192</span>.168.0.1<span class="p">;</span> <span class="m">127</span>.0.0.1<span class="p">;</span> <span class="o">}</span><span class="p">;</span>
<span class="o">}</span><span class="p">;</span>
</pre></div>
<p>Some of the fun scripts require a <span class="caps">HTTP</span> service to serve up flipped
images and all sorts of other goodness so Apache and ImageMagick are needed:</p>
<div class="highlight"><pre><span></span>$ sudo apt-get install apache2
<span class="nv">$sudo</span> apt-get -y install imagemagick
</pre></div>
<p>The last service is Squid caching proxy. Install version 3 was installed
from the repositories:</p>
<div class="highlight"><pre><span></span>$ sudo apt-get install squid3
</pre></div>
<p>Edit the Squid configuration, this is a default configuration but the
<strong>acl</strong> for the clients has been enabled along with <strong>interception</strong>
mode (read transparent) and finally call the script via
<strong>url_rewrite_program</strong>:</p>
<div class="highlight"><pre><span></span>$ grep ^<span class="o">[</span>^#<span class="o">]</span> /etc/squid3/squid.conf
acl manager proto cache_object
acl localhost src <span class="m">127</span>.0.0.1/32 ::1
acl to_localhost dst <span class="m">127</span>.0.0.0/8 <span class="m">0</span>.0.0.0/32 ::1
acl localnet src <span class="m">192</span>.168.0.0/16 <span class="c1"># RFC1918 possible internal network</span>
acl SSL_ports port <span class="m">443</span>
acl Safe_ports port <span class="m">80</span> <span class="c1"># http</span>
acl Safe_ports port <span class="m">21</span> <span class="c1"># ftp</span>
acl Safe_ports port <span class="m">443</span> <span class="c1"># https</span>
acl Safe_ports port <span class="m">70</span> <span class="c1"># gopher</span>
acl Safe_ports port <span class="m">210</span> <span class="c1"># wais</span>
acl Safe_ports port <span class="m">1025</span>-65535 <span class="c1"># unregistered ports</span>
acl Safe_ports port <span class="m">280</span> <span class="c1"># http-mgmt</span>
acl Safe_ports port <span class="m">488</span> <span class="c1"># gss-http</span>
acl Safe_ports port <span class="m">591</span> <span class="c1"># filemaker</span>
acl Safe_ports port <span class="m">777</span> <span class="c1"># multiling http</span>
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow localhost
http_access deny all
http_port <span class="m">3128</span> intercept
hierarchy_stoplist cgi-bin ?
coredump_dir /var/spool/squid3
url_rewrite_program /home/us3r/squidScripts/flipImages.pl
refresh_pattern ^ftp: <span class="m">1440</span> <span class="m">20</span>% <span class="m">10080</span>
refresh_pattern ^gopher: <span class="m">1440</span> <span class="m">0</span>% <span class="m">1440</span>
refresh_pattern -i <span class="o">(</span>/cgi-bin/<span class="p">|</span><span class="se">\?</span><span class="o">)</span> <span class="m">0</span> <span class="m">0</span>% <span class="m">0</span>
refresh_pattern . <span class="m">0</span> <span class="m">20</span>% <span class="m">4320</span>
</pre></div>
<p>Execute the following to create some protection from the subnet being
advertised and furthermore forces all of the web request to use the
Squid cache. The rule-set is by no means perfect or definitive, feel
free to tailor to your needs and provide feedback.</p>
<div class="highlight"><pre><span></span>$ grep ^<span class="o">[</span>^#<span class="o">]</span> fw-script
<span class="nv">PATH</span><span class="o">=</span>/sbin
iptables -F
iptables -t nat -F
iptables -t mangle -F
iptables -X
iptables --policy INPUT DROP
iptables --policy OUTPUT DROP
iptables --policy FORWARD DROP
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A INPUT -i eth2 -p tcp --dport <span class="m">3128</span> -j ACCEPT
iptables -A INPUT -i eth2 -p tcp --dport <span class="m">443</span> -j ACCEPT
iptables -A INPUT -i eth2 -p tcp --dport <span class="m">80</span> -j ACCEPT
iptables -A INPUT -i eth2 -p udp --dport <span class="m">53</span> -j ACCEPT
iptables -A INPUT -i eth2 -p udp --dport <span class="m">67</span> -j ACCEPT
iptables -A OUTPUT -o eth2 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i eth1 -p tcp --dport <span class="m">22</span> -j ACCEPT
iptables -A INPUT -i eth1 -p tcp --dport <span class="m">8000</span> -j ACCEPT
iptables -A INPUT -i eth1 -p udp --dport <span class="m">68</span> -j ACCEPT
iptables -A INPUT -i eth1 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -o eth1 -p tcp --dport <span class="m">80</span> -j ACCEPT
iptables -A OUTPUT -o eth1 -p udp --dport <span class="m">67</span> -j ACCEPT
iptables -A OUTPUT -o eth1 -p udp --dport <span class="m">53</span> -j ACCEPT
iptables -A OUTPUT -o eth1 -p udp --dport <span class="m">443</span> -j ACCEPT
iptables -A OUTPUT -o eth1 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -I INPUT -j LOG --log-prefix <span class="s2">"iptables denied: "</span> --log-level <span class="m">7</span>
iptables -I OUTPUT -j LOG --log-prefix <span class="s2">"iptables denied: "</span> --log-level <span class="m">7</span>
iptables -t nat -A PREROUTING -i eth2 -p tcp --dport <span class="m">80</span> -j DNAT --to-destination <span class="m">192</span>.168.0.1:3128
iptables -t nat -A POSTROUTING -o eth2 -s <span class="m">192</span>.168.0.0/24 -d <span class="m">192</span>.168.0.1 -j SNAT --to <span class="m">192</span>.168.0.1
iptables -t nat -A PREROUTING -i eth2 -p tcp --dport <span class="m">80</span> -j REDIRECT --to-port <span class="m">3128</span>
</pre></div>
<p>You can down pull down a script from the Google code repository
mentioned above which you have referenced in the Squid configuration.
There are variables in the top of the scripts that you downloaded
earlier. The variables need to be updated to reflect your system. A few
Perl module prerequisites are also listed in the top of said scripts,
access <span class="caps">CPAN</span> and install them:</p>
<div class="highlight"><pre><span></span>$ sudo perl -MCPAN -e shell
</pre></div>
<p>After the required Perl modules are installed, you should be able to
place a client on the guest network and they will retrieve sites,
although it will not take long for to notice that in this case all of
the images are inverted. Do not forget to checkout the other scripts.</p>
<p><a href="https://www.rsreese.com/assets/ternet-pinterest.png"><img alt="ternet-pinterest-scaled" src="https://www.rsreese.com/assets/ternet-pinterest-scaled.png"></a></p>
<p>Lots of fun! If I missed something or you have some feedback, use the
comment form below.</p>Block Command and Control requests using ASA 55002011-12-10T02:26:00-05:002011-12-10T02:26:00-05:00Stephen Reesetag:www.rsreese.com,2011-12-10:/block-command-and-control-requests-using-asa-5500/I recently came across a blog post demonstrating how to use the Emerging Threats rule sets in order to block malware calls to command and control (C&C) hosts. Using the script referenced in the blog post may work fine, but I want to review and update when I feel …<p>I recently came across a <a href="http://packetpushers.net/netting-the-botnets-with-cisco-asa-without-a-license/">blog</a> post demonstrating how to use the
<a href="http://rules.emergingthreats.net/fwrules/">Emerging Threats</a> rule sets in order to block malware calls to
command and control (C&C) hosts. Using the script referenced in the blog
post may work fine, but I want to review and update when I feel like it
via <span class="caps">SSH</span>. Per the Emerging Threats <a href="http://doc.emergingthreats.net/bin/view/Main/EmergingFirewallRules">wiki</a> these rules probably only
need to be updated once a week but <span class="caps">YMMV</span>.</p>
<p>Setup the <span class="caps">ASA</span> (one time):</p>
<div class="highlight"><pre><span></span>configure terminal
access-list dynamic-filter_acl extended permit ip any any
dynamic-filter enable interface outside classify-list dynamic-filter_acl
dynamic-filter drop blacklist interface outside
dynamic-filter blacklist
</pre></div>
<p>Download the C&C list from Emerging Threats:</p>
<div class="highlight"><pre><span></span>$ wget http://rules.emergingthreats.net/fwrules/emerging-PIX-CC.rules
</pre></div>
<p>Convert the list to the required format:</p>
<div class="highlight"><pre><span></span>$ sed <span class="s1">'s/ET-drop/ET-cc/g'</span> emerging-PIX-CC.rules <span class="p">|</span> egrep <span class="s2">"^access-list ET-cc deny"</span>
emerging-PIX-CC.rules <span class="p">|</span> sed <span class="s1">'s/access-list ET-cc deny ip/address/g;s/host //g;s/any </span>
<span class="s1">/255.255.255.255/g'</span> <span class="p">|</span> awk <span class="s1">'{print $1,$2,$3}'</span> > emerging-PIX-CC.rules.asa
</pre></div>
<p>Paste the list using Putty or similar. At current there are around 3000
rules so it takes a minute:</p>
<div class="highlight"><pre><span></span>configure terminal
no dynamic-filter blacklist
blacklist dynamic-filter blacklist
address x.x.x.x y.y.y.y
</pre></div>
<p>Finally, it is important to note that there could be performance
implications with implementing too many rules. Be warned you may shun
legitimate sites on shared hosting providers and the like.</p>Amazon S3 Server-Side Encryption using GSUtil2011-10-29T14:39:00-04:002011-10-29T14:39:00-04:00Stephen Reesetag:www.rsreese.com,2011-10-29:/amazon-s3-server-side-encryption-using-gsutil/If you would like to enable server-side encryption which is a relatively new feature for your Amazon S3 data using GSUtil then you need specify the header value when pushing files to their cloud. $ gsutil -h "x-amz-server-side-encryption: AES256" cp /backups/files* s3://bucket Note that server-side encryption protects your data …<p>If you would like to enable <a href="http://docs.amazonwebservices.com/AmazonS3/latest/dev/UsingServerSideEncryption.html">server-side encryption</a> which is a
relatively new <a href="http://aws.typepad.com/aws/2011/10/new-amazon-s3-server-side-encryption.html">feature</a> for your Amazon S3 data using GSUtil then you
need specify the header value when pushing files to their cloud.</p>
<div class="highlight"><pre><span></span>$ gsutil -h <span class="s2">"x-amz-server-side-encryption: AES256"</span> cp /backups/files* s3://bucket
</pre></div>
<p>Note that server-side encryption protects your data at rest and that
Amazon is managing the keys on your behalf by default, see post
<a href="http://alan.blog-city.com/amazon_s3_encryption.htm">here</a>. A better practice is to provide the encryption and decryption
before and after you send and receive your data from S3.</p>Block IRC and other communications with McAfee VirusScan2011-10-15T05:01:00-04:002011-10-15T05:01:00-04:00Stephen Reesetag:www.rsreese.com,2011-10-15:/block-irc-and-other-communications-with-mcafee-virusscan/After seeing some suspicious activitiy in my McAfee antivirus logs, I learned the Access Protection functionality, specifically IRC communication setting may be able to thwart some of the aforementioned activity. There are a number of useful setting to log or even block attempts that are not enable by default. A …<p>After seeing some suspicious activitiy in my McAfee antivirus logs, I learned the Access Protection functionality, specifically <span class="caps">IRC</span> communication setting may be able to thwart some of the aforementioned activity. There are a number of useful setting to log or even block attempts that are not enable by default. A test environment was setup using a <span class="caps">IRC</span> daemon on <a href="http://zeltser.com/remnux/">Remnux</a> and a <a href="http://nmap.org">Nmap</a> plug-in called <a href="http://nmap.org/svn/scripts/irc-info.nse">irc-info.nse</a>. An initial baseline scan/connect is made to confirm that the service residing in the virtual guest was working as advertised.</p>
<p><img alt="image" src="https://www.rsreese.com/assets/scan_win.png"></p>
<p>The host indeed had a <span class="caps">IRC</span> server running. We do not want our host communicating with <span class="caps">IRC</span> daemons so we can leverage McAfee to help us block the communication attempts. First, open up the Auto Protect settings in the VirusScan console.</p>
<p><img alt="image" src="https://www.rsreese.com/assets/vs0.png"></p>
<p>Next, “Prevent <span class="caps">IRC</span> communication” was enabled. This hosts processesshould not be making outgoing requests. If there were such requests from a process it could be indicative of malicious software contacting a C&C.</p>
<p><img alt="image" src="https://www.rsreese.com/assets/vs1.png"></p>
<p>Now the policy is being enforced, we again test the ability to connect the remote hosts <span class="caps">IRC</span> service.</p>
<p><img alt="image" src="https://www.rsreese.com/assets/scan_fail.png"></p>
<p>Nmap is able to elicit responses from the host but is unable to complete a connection to interact with the <span class="caps">IRC</span> server. The last screen shot depicts log entries reporting, and a blocking and reporting entry.</p>
<p><img alt="image" src="https://www.rsreese.com/assets/vs_log.png"></p>
<p>Be cautious of shunning all processes for a specific check as some applications may inadvertently use a port that a malicious process would typically use. Instead, consider white-listing those or one selecting known evil.</p>Variance in rwfilter results from netflow v5 and YaF2011-10-03T14:04:00-04:002011-10-03T14:04:00-04:00Stephen Reesetag:www.rsreese.com,2011-10-03:/variance-in-rwfilter-results-from-netflow-v5-and-yaf/Looking over some netflow data I notice some variance between the two sensors. Sensor s0 is v5 netflow data from a Cisco switch, s1 is from a network tap listening between a Router on a Stick and said Cisco switch. The latter is a capture from YaF listening on a …<p>Looking over some netflow data I notice some variance between the two
sensors. Sensor s0 is v5 netflow data from a Cisco switch, s1 is from a
network tap listening between a Router on a Stick and said Cisco switch.
The latter is a capture from YaF listening on a promiscuous network
interface. I needed some data so a movie streaming took care of this for
me. Here is the first difference between the two data sources.</p>
<div class="highlight"><pre><span></span>$ rwfilter --start-date<span class="o">=</span><span class="nv">$today</span> --end-date<span class="o">=</span><span class="nv">$today</span> --proto<span class="o">=</span><span class="m">0</span>-255 --pass<span class="o">=</span>stdout --sensor<span class="o">=</span>s0 <span class="p">|</span> rwstats --protocol --top --count<span class="o">=</span><span class="m">5</span> --flows
INPUT: <span class="m">675</span> Records <span class="k">for</span> <span class="m">1</span> Bin and <span class="m">675</span> Total Records
OUTPUT: Top <span class="m">5</span> Bins by Records
pro<span class="p">|</span> Records<span class="p">|</span> %Records<span class="p">|</span> cumul_%<span class="p">|</span>
<span class="m">6</span><span class="p">|</span> <span class="m">675</span><span class="p">|</span><span class="m">100</span>.000000<span class="p">|</span><span class="m">100</span>.000000<span class="p">|</span>
$ rwfilter --start-date<span class="o">=</span><span class="nv">$today</span> --end-date<span class="o">=</span><span class="nv">$today</span> --proto<span class="o">=</span><span class="m">0</span>-255 --pass<span class="o">=</span>stdout --sensor<span class="o">=</span>s1 <span class="p">|</span> rwstats --protocol --top --count<span class="o">=</span><span class="m">5</span> --flows
INPUT: <span class="m">2640</span> Records <span class="k">for</span> <span class="m">3</span> Bins and <span class="m">2640</span> Total Records
OUTPUT: Top <span class="m">5</span> Bins by Records
pro<span class="p">|</span> Records<span class="p">|</span> %Records<span class="p">|</span> cumul_%<span class="p">|</span>
<span class="m">17</span><span class="p">|</span> <span class="m">1927</span><span class="p">|</span> <span class="m">72</span>.992424<span class="p">|</span> <span class="m">72</span>.992424<span class="p">|</span>
<span class="m">6</span><span class="p">|</span> <span class="m">712</span><span class="p">|</span> <span class="m">26</span>.969697<span class="p">|</span> <span class="m">99</span>.962121<span class="p">|</span>
<span class="m">1</span><span class="p">|</span> <span class="m">1</span><span class="p">|</span> <span class="m">0</span>.037879<span class="p">|</span><span class="m">100</span>.000000<span class="p">|</span>
</pre></div>
<p>The difference between the flow data here is the v5 data only shows <span class="caps">TCP</span>
connections at this point where as the tap is seeing <span class="caps">ICMP</span>, <span class="caps">TCP</span> and <span class="caps">UDP</span>.
The next set of queries are from a streaming movie which the output has
been cut for brevity.</p>
<div class="highlight"><pre><span></span>$ rwfilter --start-date<span class="o">=</span><span class="nv">$today</span> --end-date<span class="o">=</span><span class="nv">$today</span> --sensor<span class="o">=</span>s0 --type<span class="o">=</span>all --proto<span class="o">=</span><span class="m">1</span>,6,17 --pass<span class="o">=</span>stdout --daddress<span class="o">=</span><span class="m">172</span>.16.0.10 <span class="p">|</span> rwsort --fields<span class="o">=</span>bytes <span class="p">|</span> rwcut --fields<span class="o">=</span>sip,sport,dip,dport,bytes
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span><span class="m">65184</span><span class="p">|</span> <span class="m">57713601</span><span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span><span class="m">65183</span><span class="p">|</span> <span class="m">58666986</span><span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span><span class="m">65183</span><span class="p">|</span> <span class="m">146904926</span><span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span><span class="m">65184</span><span class="p">|</span> <span class="m">153098218</span><span class="p">|</span>
$ rwfilter --start-date<span class="o">=</span><span class="nv">$today</span> --end-date<span class="o">=</span><span class="nv">$today</span> --sensor<span class="o">=</span>s1 --type<span class="o">=</span>all --proto<span class="o">=</span><span class="m">1</span>,6,17 --pass<span class="o">=</span>stdout --daddress<span class="o">=</span><span class="m">172</span>.16.0.10 <span class="p">|</span> rwsort --fields<span class="o">=</span>bytes <span class="p">|</span> rwcut --fields<span class="o">=</span>sip,sport,dip,dport,bytes
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span><span class="m">65183</span><span class="p">|</span> <span class="m">110759034</span><span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span><span class="m">65184</span><span class="p">|</span> <span class="m">111370758</span><span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span><span class="m">65183</span><span class="p">|</span> <span class="m">148760315</span><span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span><span class="m">65184</span><span class="p">|</span> <span class="m">150597449</span><span class="p">|</span>
</pre></div>
<p>The item to note here is the v5 netflow is reporting more bytes than the
network tap for similar source and <span class="caps">IP</span> addresses for the respective
destination <span class="caps">IP</span> addresses. Same results with the next filter.</p>
<div class="highlight"><pre><span></span>$ rwfilter --start-date<span class="o">=</span><span class="nv">$today</span> --end-date<span class="o">=</span><span class="nv">$today</span> --protocol<span class="o">=</span><span class="m">1</span>,6,17 --sensor<span class="o">=</span>s0 --type<span class="o">=</span>all --pass<span class="o">=</span>stdout --saddress<span class="o">=</span><span class="m">69</span>.241.37.66 --daddress<span class="o">=</span><span class="m">172</span>.16.0.10 <span class="p">|</span> rwstats --count<span class="o">=</span><span class="m">10</span> --fields<span class="o">=</span>sip,dip,scc,bytes,sport
INPUT: <span class="m">4</span> Records <span class="k">for</span> <span class="m">4</span> Bins and <span class="m">4</span> Total Records
OUTPUT: Top <span class="m">10</span> Bins by Records
sIP<span class="p">|</span> dIP<span class="p">|</span>scc<span class="p">|</span> bytes<span class="p">|</span>sPort<span class="p">|</span> Records<span class="p">|</span> %Records<span class="p">|</span> cumul_%<span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span> us<span class="p">|</span> <span class="m">111370758</span><span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">1</span><span class="p">|</span> <span class="m">25</span>.000000<span class="p">|</span> <span class="m">25</span>.000000<span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span> us<span class="p">|</span> <span class="m">150597449</span><span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">1</span><span class="p">|</span> <span class="m">25</span>.000000<span class="p">|</span> <span class="m">50</span>.000000<span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span> us<span class="p">|</span> <span class="m">110759034</span><span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">1</span><span class="p">|</span> <span class="m">25</span>.000000<span class="p">|</span> <span class="m">75</span>.000000<span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span> us<span class="p">|</span> <span class="m">148760315</span><span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">1</span><span class="p">|</span> <span class="m">25</span>.000000<span class="p">|</span><span class="m">100</span>.000000<span class="p">|</span>
$ rwfilter --start-date<span class="o">=</span><span class="nv">$today</span> --end-date<span class="o">=</span><span class="nv">$today</span> --protocol<span class="o">=</span><span class="m">1</span>,6,17 --sensor<span class="o">=</span>s1 --type<span class="o">=</span>all --pass<span class="o">=</span>stdout --saddress<span class="o">=</span><span class="m">69</span>.241.37.66 --daddress<span class="o">=</span><span class="m">172</span>.16.0.10 <span class="p">|</span> rwstats --count<span class="o">=</span><span class="m">10</span> --fields<span class="o">=</span>sip,dip,scc,bytes,sport
INPUT: <span class="m">4</span> Records <span class="k">for</span> <span class="m">4</span> Bins and <span class="m">4</span> Total Records
OUTPUT: Top <span class="m">10</span> Bins by Records
sIP<span class="p">|</span> dIP<span class="p">|</span>scc<span class="p">|</span> bytes<span class="p">|</span>sPort<span class="p">|</span> Records<span class="p">|</span> %Records<span class="p">|</span> cumul_%<span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span> us<span class="p">|</span> <span class="m">57713601</span><span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">1</span><span class="p">|</span> <span class="m">25</span>.000000<span class="p">|</span> <span class="m">25</span>.000000<span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span> us<span class="p">|</span> <span class="m">153098218</span><span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">1</span><span class="p">|</span> <span class="m">25</span>.000000<span class="p">|</span> <span class="m">50</span>.000000<span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span> us<span class="p">|</span> <span class="m">146904926</span><span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">1</span><span class="p">|</span> <span class="m">25</span>.000000<span class="p">|</span> <span class="m">75</span>.000000<span class="p">|</span>
<span class="m">69</span>.241.37.66<span class="p">|</span> <span class="m">172</span>.16.0.10<span class="p">|</span> us<span class="p">|</span> <span class="m">58666986</span><span class="p">|</span> <span class="m">80</span><span class="p">|</span> <span class="m">1</span><span class="p">|</span> <span class="m">25</span>.000000<span class="p">|</span><span class="m">100</span>.000000<span class="p">|</span>
</pre></div>
<p>The output difference between the two sensors are minimal in most cases
and a large portion could be due to traffic that the tap may have better
insight to report though more analysis needs to be done using tcpdump or
Wireshark. Nevertheless this should be considered when determine the
senor requirements and the type of data that you would like to view
reporting for. That said, any reporting is be better than none.</p>Configure YAF on Linux for NetFlow collection from a network tap or SPAN2011-08-26T21:27:00-04:002011-08-26T21:27:00-04:00Stephen Reesetag:www.rsreese.com,2011-08-26:/configure-yaf-on-linux-for-netflow-collection-from-a-network-tap-or-span/In a previous post SiLK was setup on a Debian host using NetFlow v5 from a Cisco switch. This worked well but I also have a network tap and said Cisco switch is capable of capturing data via SPAN port(s). This got me thinking about what difference I may …<p>In a previous <a href="https://www.rsreese.com/configure-silk-on-linux-for-netflow-collection-from-a-cisco-router/">post</a> SiLK was setup on a Debian host using NetFlow v5
from a Cisco switch. This worked well but I also have a network tap and
said Cisco switch is capable of capturing data via <span class="caps">SPAN</span> port(s). This
got me thinking about what difference I may see between the two NetFlow
sources. This guide walks through setting up <span class="caps">YAF</span> on a Debian Linux host
to receive data from a network tap or Switched Port Analyzer (<span class="caps">SPAN</span>) and
converting it using <a href="http://tools.netsa.cert.org/yaf/index.html">Yet Another Flowmeter (<span class="caps">YAF</span>)</a>.</p>
<p>First, your host will need to obtain data from your network tap or <span class="caps">SPAN</span>
port. I have two network interface cards in my box so I connected the
non-management interface to the tap and started the interface without an
<span class="caps">IP</span> in promiscuous mode. If you would like to use a <span class="caps">SPAN</span> port seek
guidance <a href="https://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008015c612.shtml">here</a>.</p>
<p>Note that this guide assumes that you already have compiled and
successfully built SiLK. If not checkout this [post][].</p>
<p>You first need <em>libfixbuf</em> - <a href="http://tools.netsa.cert.org/fixbuf/index.html"><span class="caps">IPFIX</span> Protocol Library</a>. Before building
<span class="caps">IPFIX</span> will need glib2 and its respective development libraries, I did
not have the latter so a little <span class="caps">APT</span> action takes care of that for me.</p>
<div class="highlight"><pre><span></span>$ sudo apt-get install libglib2-dev
</pre></div>
<p>Building libfixbuf is straigtforward once the prerequetes are in place.</p>
<div class="highlight"><pre><span></span>$ ./configure --prefix<span class="o">=</span>/usr
$ make
$ make install
</pre></div>
<p>Next we are going to build <a href="http://tools.netsa.cert.org/yaf/index.html"><span class="caps">YAF</span> is Yet Another Flowmeter</a> which has several prerequisites. <em>libpcap</em> needs to be
installed along with its respective development libraries. I also
installed the required <em><span class="caps">PCRE</span></em> required libraries for application labeling.</p>
<div class="highlight"><pre><span></span>$ sudo apt-get install libpcap-dev
$ sudo apt-get install libpcre3-dev
</pre></div>
<p>Next we can build <span class="caps">YAF</span>.</p>
<div class="highlight"><pre><span></span>$ ./configure --prefix<span class="o">=</span>/usr --enable-applabel
$ make
$ sudo make install
</pre></div>
<p>Now that everything is ready to go we have a little housekeeping to do
on the <span class="caps">YAF</span> configuration files. I placed the <span class="caps">YAF</span> configuration file in
<em>/etc/silk/yaf.conf</em>. This file contains all of the setting such as
which interface to listen on, <span class="caps">IPFIX</span> port, etc.<em><br>
</em></p>
<div class="highlight"><pre><span></span>## ------------------------------------------------------------------------
## yaf.conf
## YAF daemon startup script configuration file
## ------------------------------------------------------------------------
## Copyright (C) 2007-2011 Carnegie Mellon University. All Rights Reserved.
## ------------------------------------------------------------------------
## Authors: Brian Trammell
## ------------------------------------------------------------------------
## GNU General Public License (GPL) Rights pursuant to Version 2, June 1991
## Government Purpose License Rights (GPLR) pursuant to DFARS 252.227-7013
## ------------------------------------------------------------------------
# Must be non-empty to start YAF
ENABLED=yes
##### Capture Options ##########################################################
# Live capture type. Must be pcap, or dag for Endace DAG if YAF was built
# with libdag.
YAF_CAP_TYPE=pcap
# Live capture interface name.
YAF_CAP_IF=eth0
##### Export Options ###########################################################
# IPFIX transport protocol to use for export. Must be one of tcp or udp, or
# sctp if fixbuf was built with SCTP support.
YAF_IPFIX_PROTO=tcp
# Hostname or IP address of IPFIX collector to export flows to.
YAF_IPFIX_HOST=localhost
# If present, connect to the IPFIX collector on the specified port.
# Defaults to port 4739, the IANA-assigned port for IPFIX
YAF_IPFIX_PORT=18000
##### Logging and State Options ################################################
# Path to state location directory; contains the log and pidfiles unless
# modified by the following configuration parameters.
# Defaults to <span class="cp">${</span><span class="n">prefix</span><span class="cp">}</span>/var.
#YAF_STATEDIR=
# Path to PID file for YAF. Defaults to YAF_STATEDIR/yaf.pid
#YAF_PIDFILE=
# File or syslog facility name for YAF logging. If file, must be an absolute
# path to a logfile. Defaults to YAF_STATEDIR/yaf.log
#YAF_LOG=
# File or syslog facility name for YAF airdaemon logging. If file, must be an
# absolute path to a logfile. Defaults to YAF_STATEDIR/airdaemon-yaf.log
#YAF_DAEMON_LOG=
##### Miscellaneous Options ####################################################
# If present, become the specified user after starting YAF
#YAF_USER=
# Additional flags to pass to the YAF process. Use --silk --ip4-only for
# export to SiLK rwflowpack or SiLK flowcap.
YAF_EXTRAFLAGS="--silk"
</pre></div>
<p>Made sure there was a sensor definition in the <em>/netflow/silk.conf</em>.</p>
<div class="highlight"><pre><span></span><span class="nx">sensor</span> <span class="mi">0</span> <span class="nx">s0</span> <span class="s2">"v5 netflow from router"</span>
<span class="nx">sensor</span> <span class="mi">1</span> <span class="nx">s1</span> <span class="s2">"YAF converted from tap"</span>
<span class="kr">class</span> <span class="nx">all</span>
<span class="nx">sensors</span> <span class="nx">s0</span> <span class="nx">s1</span>
<span class="nx">end</span> <span class="kr">class</span>
</pre></div>
<p>The <em>/etc/silk/sensor.conf</em> configuration file also need to be updated
with the new sensor definition. In this case s1 is our tap.</p>
<div class="highlight"><pre><span></span>probe s0 netflow-v5
listen-on-port 9990
protocol udp
accept-from-host 172.16.0.1
end probe
sensor s0
netflow-v5-probes s0
internal-ipblocks 172.16.0.0/24
external-ipblocks remainder
end sensor
probe s1 ipfix
listen-on-port 18000
protocol tcp
accept-from-host 127.0.0.1
end probe
sensor s1
ipfix-probes s1
internal-ipblocks 172.16.0.0/24
external-ipblocks remainder
end sensor
</pre></div>
<p>Lastly, start <span class="caps">YAF</span> assuming that you have rwflowpack running from the
SiLK package per the previous [post][].</p>
<div class="highlight"><pre><span></span>$ sudo yaf --silk --ipfix<span class="o">=</span>tcp --live<span class="o">=</span>pcap --in<span class="o">=</span>eth0 --out<span class="o">=</span><span class="m">127</span>.0.0.1 --ipfix-port<span class="o">=</span><span class="m">18000</span> <span class="p">&</span>
</pre></div>
<p>You should now be capturing data and converting into a format that SiLK
can process via <span class="caps">YAF</span>.</p>Configure SiLK on Linux for NetFlow collection from a Cisco router2011-08-15T00:43:00-04:002011-08-15T00:43:00-04:00Stephen Reesetag:www.rsreese.com,2011-08-15:/configure-silk-on-linux-for-netflow-collection-from-a-cisco-router/This guide walks through configuring SiLK from a source install on a Debian 6 host in order to collect NetFlow data from a Cisco router. The guides here and here written by CERT NetSA are quite good but lack some detail specific to the Debian distribution which required a bit …<p>This guide walks through configuring <a href="http://tools.netsa.cert.org/silk/index.html">SiLK</a> from a source install on a
<a href="http://www.debian.org/">Debian</a> 6 host in order to collect <a href="https://www.cisco.com/en/US/products/ps6601/products_ios_protocol_group_home.html">NetFlow</a> data from a Cisco
router. The guides <a href="http://tools.netsa.cert.org/silk/install-handbook.html#x1-130002">here</a> and <a href="https://tools.netsa.cert.org/confluence/display/tt/Configure+SiLK+for+NetFlow+collection+from+a+Cisco+router">here</a> written by <span class="caps">CERT</span> <a href="https://www.cert.org/netsa/">NetSA</a> are
quite good but lack some detail specific to the Debian distribution
which required a bit of mucking about to get everything functioning
correctly. This assumes that you have a Cisco router to send NetFlow
data to a host on your network, in this case, a Debian host.</p>
<p>Installation:</p>
<p>First install a prerequisite.</p>
<div class="highlight"><pre><span></span>$ sudo apt-get install libpcap-dev
</pre></div>
<p>Next untar and change into the SiLK directory. For Debian I found that
using the <em>/usr</em> directory worked well. By default the configure script
uses <em>/usr/local</em> in which it places the binaries, libraries, etc
outside of Debians default paths.</p>
<div class="highlight"><pre><span></span>$ ./configure --prefix<span class="o">=</span>/usr --sysconfdir<span class="o">=</span>/etc/silk --enable-data-rootdir<span class="o">=</span>/netflow
--enable-ipv6 --enable-output-compression
</pre></div>
<p>Your output should be something along the following:</p>
<div class="highlight"><pre><span></span> * Configured package: SiLK 2.4.5
* Host type: x86_64-unknown-linux-gnu
* Source files ($top_srcdir): .
* Install directory: /usr
* Root of packed data tree: /netflow
* Packing logic: via run-time plugin
* Timezone support: UTC
* Default compression method: SK_COMPMETHOD_ZLIB
* IPv6 support: YES
* IPFIX collection support: YES (-pthread -lfixbuf -lgthread-2.0 -lrt -lglib-2.0)
* Transport encryption support: NO (gnutls not found)
* IPA support: NO
* LIBPCAP support: YES (-lpcap)
* ADNS support: NO
* Python support: NO
* Build analysis tools: YES
* Build packing tools: YES
* Compiler (CC): gcc
* Compiler flags (CFLAGS): -I$(srcdir) -I$(top_builddir)/src/include -I$(top_srcdir)/src/include -DNDEBUG -O3 -fno-strict-aliasing -Wall -W -Wmissing-prototypes -Wformat=2 -Wdeclaration-after-statement -Wpointer-arith
* Linker flags (LDFLAGS):
* Libraries (LIBS): -lz -ldl -lm
</pre></div>
<p>Lastly:</p>
<div class="highlight"><pre><span></span>$ make
$ sudo make install
</pre></div>
<p>Configuration:</p>
<p>Example files are available in the tarball that you extracted. Modified
versions or notes for Debian and similar architectures available below.</p>
<p><em>/netflow/silk.conf</em> in your data directory, the default is <em>/data</em> but
I used /netflow as you can see in the configure toggle above. The
changes I made were to reduce the number of sensors.</p>
<div class="highlight"><pre><span></span><span class="err">#</span> <span class="nx">The</span> <span class="nx">syntactic</span> <span class="nx">format</span> <span class="nx">of</span> <span class="k">this</span> <span class="nx">file</span>
<span class="err">#</span> <span class="nx">version</span> <span class="mi">2</span> <span class="nx">supports</span> <span class="nx">sensor</span> <span class="nx">descriptions</span><span class="p">,</span> <span class="nx">but</span> <span class="nx">otherwise</span> <span class="nx">identical</span> <span class="nx">to</span> <span class="mi">1</span>
<span class="nx">version</span> <span class="mi">2</span>
<span class="nx">sensor</span> <span class="mi">0</span> <span class="nx">s0</span> <span class="s2">"Description for sensor S0"</span>
<span class="nx">sensor</span> <span class="mi">1</span> <span class="nx">s1</span>
<span class="kr">class</span> <span class="nx">all</span>
<span class="nx">sensors</span> <span class="nx">s0</span> <span class="nx">s1</span>
<span class="nx">end</span> <span class="kr">class</span>
<span class="err">#</span> <span class="nx">Editing</span> <span class="nx">above</span> <span class="k">this</span> <span class="nx">line</span> <span class="nx">is</span> <span class="nx">sufficient</span> <span class="k">for</span> <span class="nx">sensor</span> <span class="nx">definition</span><span class="p">.</span>
</pre></div>
<p><em>/etc/silk/sensor.conf</em> is the definition for the data coming in from
your Cisco router:</p>
<div class="highlight"><pre><span></span>probe s0 netflow-v5
listen-on-port 9990
protocol udp
accept-from-host 172.16.0.1
end probe
sensor s0
netflow-v5-probes s0
internal-ipblocks 172.16.0.0/24
external-ipblocks remainder
end sensor
</pre></div>
<p><em>/etc/silk/rwflowpack.conf</em>:</p>
<div class="highlight"><pre><span></span>### Packer configuration file -*- sh -*-
##
## The canonical pathname for this file is /usr/local/etc/rwflowpack.conf
##
## RCSIDENT("<span class="nv">$SiLK</span>: rwflowpack.conf.in 16306 2010-09-15 18:14:41Z mthomas $")
##
## This is a /bin/sh file that gets loaded by the init.d/rwflowpack
## wrapper script, and this file must follow /bin/sh syntax rules.
# Set to non-empty value to enable rwflowpack
ENABLED=yes
# These are convenience variables for setting other values in this
# configuration file; their use is not required.
statedirectory=/var/lib/rwflowpack
# If CREATE_DIRECTORIES is set to "yes", the directories named in this
# file will be created automatically if they do not already exist
CREATE_DIRECTORIES=yes
# Full path of the directory containing the "rwflowpack" program
BIN_DIR=/usr/sbin
# The full path to the sensor configuration file. Used by
# --sensor-configuration. YOU MUST PROVIDE THIS (the value is ignored
# when INPUT_MODE is "respool").
SENSOR_CONFIG=/etc/silk/sensor.conf
# The full path to the root of the tree under which the packed SiLK
# Flow files will be written. Used by --root-directory.
DATA_ROOTDIR=/netflow
# The full path to the site configuration file. Used by
# --site-config-file. If not set, defaults to silk.conf in the
# <span class="cp">${</span><span class="n">DATA_ROOTDIR</span><span class="cp">}</span>.
SITE_CONFIG=/netflow/silk.conf
# Specify the path to the packing-logic plug-in that rwflowpack should
# load and use. The plug-in provides functions that determine into
# which class and type each flow record will be categorized and the
# format of the files that rwflowpack will write. When SiLK has been
# configured with hard-coded packing logic (i.e., when
# --enable-packing-logic was specified to the configure script), this
# value should be empty. A default value for this switch may be
# specified in the <span class="cp">${</span><span class="n">SITE_CONFIG</span><span class="cp">}</span> site configuration file. This value
# is ignored when INPUT_MODE is "respool".
PACKING_LOGIC=
# Data input mode. Valid values are:
# * "stream" mode to read from the network or from probes that have
# poll-directories
# * "fcfiles" to process flowcap files on the local disk
# * "respool" to process SiLK flow files maintaining the sensor and
# class/type values that already exist on those records.
INPUT_MODE=stream
# Directory in which to look for incoming flowcap files in "fcfiles"
# mode or for incoming SiLK files in "respool" mode
INCOMING_DIR=<span class="cp">${</span><span class="n">statedirectory</span><span class="cp">}</span>/incoming
# Directory to move input files to after successful processing. When
# in "stream" mode, these are the files passed to any probe with a
# poll-directory directive. When in "fcfiles" mode, these are the
# flowcap files. When in "respool" mode, these are the SiLK Flow
# files. If not set, the input files are not archived but are deleted
# instead.
ARCHIVE_DIR=<span class="cp">${</span><span class="n">statedirectory</span><span class="cp">}</span>/archive
# When using the ARCHIVE_DIR, normally files are stored in
# subdirectories of the ARCHIVE_DIR. If this variable's value is 1,
# files are stored in ARCHIVE_DIR itself, not in subdirectories of it.
FLAT_ARCHIVE=0
# Directory to move an input file into if there is a problem opening
# the file. If this value is not set, rwflowpack will exit when it
# encounters a problem file. When in "fcfiles" mode, these are the
# flowcap files. When in "stream" mode, these are the files passed to
# any probe with a poll-directory directive.
ERROR_DIR= #<span class="cp">${</span><span class="n">statedirectory</span><span class="cp">}</span>/error
# Data output mode. Valid values are "local" and "remote". "local"
# writes the hourly data files to the local disk. "remote" creates
# small files (called incremental files) that must be processed by
# rwflowappend to create the hourly files.
OUTPUT_MODE=local
# Directory in which the incremental files are written when the
# OUTPUT_MODE is "remote". Typically there is an rwsender deamon that
# polls this directory for new incremental files.
SENDER_DIR=<span class="cp">${</span><span class="n">statedirectory</span><span class="cp">}</span>/sender-incoming
# Temporary directory in which to build incremental files prior to
# handing them to rwsender. Used only when OUTPUT_MODE is "remote".
INCREMENTAL_DIR=<span class="cp">${</span><span class="n">statedirectory</span><span class="cp">}</span>/incremental
# The type of compression to use for packed files. Left empty, the
# value chosen at compilation time will be used. Valid values are
# "best" and "none". Other values are system-specific (the available
# values are listed in the description of the --compression-method
# switch in the output of rwflowpack --help).
COMPRESSION_TYPE=best
# Interval between attempts to check the INCOMING_DIR or
# poll-directory probe entries for new files, in seconds. This may be
# left blank, and will default to 15.
POLLING_INTERVAL=
# Interval between periodic flushes of open SiLK Flow files to disk,
# in seconds. This may be left blank, and will default to 120.
FLUSH_TIMEOUT=
# Maximum number of SiLK Flow files to have open for writing
# simultaneously. This may be left blank, and will default to 64
FILE_CACHE_SIZE=
# Whether rwflowpack should use advisory write locks. 1=yes, 0=no.
# Set to zero if messages like "Cannot get a write lock on file"
# appear in rwflowpack's log file.
FILE_LOCKING=1
# Whether rwflowpack should include the input and output SNMP
# interfaces and the next-hop-ip in the output files. 1=yes, 0=no.
# The default is no, and these values are not stored to save disk
# space. (The input and output fields contain VLAN tags when the
# sensor.conf file contains the attribute "interface-values vlan".)
PACK_INTERFACES=0
###
# The type of logging to use. Valid values are "legacy" and "syslog".
LOG_TYPE=syslog
# The lowest level of logging to actually log. Valid values are:
# emerg, alert, crit, err, warning, notice, info, debug
LOG_LEVEL=info
# The full path of the directory where the log files will be written
# when LOG_TYPE is "legacy".
LOG_DIR=/var/log
# The full path of the directory where the PID file will be written
PID_DIR=<span class="cp">${</span><span class="n">LOG_DIR</span><span class="cp">}</span>
# The user this program runs as; root permission is required only when
# rwflowpack listens on a privileged port.
USER=root
#USER=`whoami` # run as user invoking the script
# Extra options to pass to rwflowpack
EXTRA_OPTIONS=
</pre></div>
<p><em>/etc/init.d/rwflowback</em> directory, the only change was to line 38 in
order to change to the configuration specified in the configure statement.</p>
<div class="highlight"><pre><span></span>SCRIPT_CONFIG_LOCATION="/etc/silk"
</pre></div>
<p>With everything installed in their respective locations it is time to
move on to setting up the Cisco device.</p>
<div class="highlight"><pre><span></span>Router(config)# ip cef
Router(config)# ip flow-export source Loopback0
Router(config)# ip flow-export version 5
Router(config)# ip flow-export destination x.x.x.x 9990
Router(config)# interface f1/0
Router(config-if)# ip flow ingress
Router(config-if)# ip flow egress
</pre></div>
<p>I hope this helps. If you have any comments or questions, leave a
comment below.</p>Setting Google Storage object ACL for authenticated downloads2011-07-17T03:06:00-04:002011-07-17T03:06:00-04:00Stephen Reesetag:www.rsreese.com,2011-07-17:/setting-google-storage-object-acl-for-authenticated-downloads/Google’s gsutil is a great tool for pushing, retrieving and setting permissions on objects uploaded to Google Storage. I was reviewing the documentation on the Sharing and Collaboration page, specifically the Authenticated Browser Download section and realized there were a couple of small mistakes, err typos. I wanted to …<p>Google’s <a href="http://code.google.com/apis/storage/docs/gsutil.html">gsutil</a> is a great tool for pushing, retrieving and setting
permissions on objects uploaded to Google Storage. I was reviewing the
documentation on the <a href="https://code.google.com/apis/storage/docs/collaboration.html#browser">Sharing and Collaboration</a> page, specifically
the <em>Authenticated Browser Download</em> section and realized there were a
couple of small mistakes, err typos. I wanted to give someone read
privileges to an object via their email address. The correct format is
posted in this <a href="http://pastebin.com/3KFKwnVm">Paste</a>.</p>
<p>The <strong>EmailAddress</strong> tag needs to be closed and the <strong>Permission</strong> tags
need to be moved outside of the <strong>Scope</strong> tag. With all of this said I
later came across <a href="https://code.google.com/apis/storage/docs/accesscontrol.html">Access Control</a> page which is documented correctly.
Go figure.</p>Running NIX Retina and Nessus vulnerability scans with least privileges2011-06-17T02:46:00-04:002011-06-17T02:46:00-04:00Stephen Reesetag:www.rsreese.com,2011-06-17:/running-nix-retina-and-nessus-vulnerability-scans-with-least-privileges/When you are running those vulnerability scans of Linux and UNIX hosts I hope that you are following best practices for keeping a host secure during the process. Both Retina and Nessus rely upon SSH in order to connect to a remote host and run a number of commands to …<p>When you are running those vulnerability scans of Linux and <span class="caps">UNIX</span> hosts I
hope that you are following best practices for keeping a host secure
during the process. Both <a href="http://www.eeye.com/Products/Retina/Network-Security-Scanner.aspx">Retina</a> and <a href="http://www.tenable.com/products/nessus">Nessus</a> rely upon <span class="caps">SSH</span> in
order to connect to a remote host and run a number of commands to
compare the querys to their respective databases of known issues, vulns
and configuration faults. Removing the directive in the sshd_config
file to enable root login is definitely not best practice, and is
borderline “hacking naked”. Lucky for us both Tenable and eEye have
documented the methods for running scans with su or sudo (the latter preferred).</p>
<ul>
<li><a href="http://www.eeye.com/Support/Knowledge-Base/Article.aspx?id=KB000883">Retina: How to enable <span class="caps">SUDO</span> support for Retina</a></li>
<li><a href="http://blog.tenablesecurity.com/2010/05/nessus-spotlight-susudo-feature.html">Nessus Spotlight: su+sudo Feature</a></li>
</ul>
<p>As the Retina publication states, you may want to limit the commands
that the sudo user may run. To do this you can look at the Retina logs
on your Windows client; or after a successful scan with take a peek at
the <span class="caps">NIX</span> user history in order to determine what commands were run. This
could also be useful for scripting up a self-scan for a host that may
lack a <span class="caps">SSH</span> service. Another method may beside reviewing the scanners
logs might be to check the history of the secure or messages log to
determine what commands were run and successfully returned a response.</p>
<p>After determining what commands the host needs to correctly run a
credentialed scan you can limit the users <em>sudo</em> privileges in the
<em>/etc/sudoers</em> file. This allows users bob and alice to execut cmd0,
cmd1 and cmdn, though disables su and the ability to change to a shell
that may not log correctly.</p>
<div class="highlight"><pre><span></span>Cmnd_Alias SHELLS = /usr/bin/sh, /usr/bin/csh,
/usr/bin/ksh, /usr/local/bin/tcsh,
/usr/bin/rsh, /usr/local/bin/zsh
Cmnd_Alias RETINA = /usr/sbin/cmd0, /usr/sbin/cmd1, /usr/sbin/cmdn
User_Alias RETINA_USERS = alice, bob
RETINA_USERS ALL = !/usr/bin/su, !SHELLS, RETINA
</pre></div>
<p>As usual, <span class="caps">YMMV</span> so let me know if this is helpful or misinforming.</p>Use Facebook CDN to host website photo gallerys2011-04-19T23:23:00-04:002011-04-19T23:23:00-04:00Stephen Reesetag:www.rsreese.com,2011-04-19:/use-facebook-cdn-to-host-website-photo-gallerys/I was thinking about how to retrieve photos from Facebook photo gallery’s and came across a number of solutions. Most of the solutions were for blog or CMS and furthermore required caching your credentials in a database along with a few other hoops in order to access your albums …<p>I was thinking about how to retrieve photos from Facebook photo
gallery’s and came across a number of solutions. Most of the solutions
were for blog or <span class="caps">CMS</span> and furthermore required caching your credentials
in a database along with a few other hoops in order to access your
albums and display them on a third party site. I thought this was a bit
odd as if you want to share photos on your blog or site you should be
able to just make the album public and use Facebooks <span class="caps">API</span> to connect
since they are going to be public at that point. After poking around
this ended up being much easier than expected and it works with Facebook
Fan Pages which is where I think this would be most useful.</p>
<ol>
<li>You need to create a Facebook <a href="http://developers.facebook.com/">application account</a> which will
provide you with your <strong>appId</strong> and <strong>secret</strong>.</li>
<li>Next you need to get the <span class="caps">PHP</span> <span class="caps">SDK</span> from <a href="https://github.com/facebook/php-sdk/">GitHub</a>. All you need is
the facebook.php page but feel free to grab the <span class="caps">ZIP</span> and explore.
There is an example to experiment with.</li>
<li>Lastly you can use the <a href="https://code.google.com/p/reese/source/browse/trunk/facebook-cdn-photo-gallery.php">code</a> provided on <a href="https://code.google.com/">Google Code</a> as a
basic start to implementing a photo gallery on your site.</li>
</ol>
<p>The code displays thumbnails, source images along with name’s
(caption’s) below each image that have them and finally the source which
you can use to derive other goodies that you might want to use in you
gallery. Some examples are different size thumbnails, id, comments, etc.</p>
<p>*Note that the script does not parse double quotes in photo captions
well at this point.</p>
<p>If you notice any issues, room for improvement or features feel free to
leave a comment or post an issue over at the Google Code page.</p>Blocking evil with the Enhanced Mitigation Experience Toolkit EMET2011-01-29T03:18:00-05:002011-01-29T03:18:00-05:00Stephen Reesetag:www.rsreese.com,2011-01-29:/blocking-evil-with-the-enhanced-mitigation-experience-toolkit-emet/While experimenting with EMET I decided to put together a little presentation demonstrating how it can be used to prevent exploitation of a known threat to Acrobat Reader. The presentation first demonstrates the exploit using Metasploit, provides some high level analysis and then goes on to describe how EMET can …<p>While experimenting with <a href="https://www.microsoft.com/downloads/en/details.aspx?FamilyID=c6f0a6ee-05ac-4eb6-acd0-362559fd2f04"><span class="caps">EMET</span></a> I decided to put together a little
presentation demonstrating how it can be used to prevent exploitation of
a known threat to Acrobat Reader. The presentation first demonstrates
the exploit using Metasploit, provides some high level analysis and then
goes on to describe how <span class="caps">EMET</span> can mitigate the vulnerability. It may be a
little choppy to follow so feel free to provide any constructive
feedback. The presentation is available via <a href="https://www.rsreese.com/assets/EMET_Reese_presentation_v5.pdf"><span class="caps">PDF</span></a>.</p>Pseudo Gmail address obfuscation2011-01-10T02:35:00-05:002011-01-10T02:35:00-05:00Stephen Reesetag:www.rsreese.com,2011-01-10:/pseudo-gmail-address-obfuscation/I was hunting around for a way to create email aliases for mailing-lists and whatnot. It is a little disappointing to learn that there is not away to create true aliases with Google’s Gmail. You can create aliases if using Google’s hosted application service but I do not …<p>I was hunting around for a way to create email aliases for mailing-lists
and whatnot. It is a little disappointing to learn that there is not
away to create true aliases with Google’s Gmail. You can create aliases
if using Google’s hosted application service but I do not use this for
my personal mail. Here are three interesting item’s I came across;
Google’s mail servers ignore period’s for the username context,
googlemail.com may be used instead of gmail.com and finally you can
append notes after a plus symbol.</p>
<p><code>firstname.lastname@gmail.com</code> may be written as
<code>first.name.last.name@gmail.com</code></p>
<p><code>firstname.lastname@gmail.com</code> may be written as
<code>firstname.lastname@googlemail.com</code></p>
<p><code>firstname.lastname@gmail.com</code> may be written as
<code>firstname.lastname+sometext@gmail.com</code></p>
<p>It’s not really obfuscation but it may help confuse someone not the
wiser. Maybe one day Gmail will allow for true alias creation.</p>Insecure Library Loading Could Allow Remote Code Execution2010-11-23T21:26:00-05:002010-11-23T21:26:00-05:00Stephen Reesetag:www.rsreese.com,2010-11-23:/insecure-library-loading-could-allow-remote-code-execution/Note this is an older post that I am migrating from another blog I previously maintained. Metasploit has already provide a nice write up of the pwning, I mean testing the vector http://blog.metasploit.com/2010/08/exploiting-dll-hijacking-flaws.html. It does involve a bit of prep work but I …<p><em>Note this is an older post that I am migrating from another blog I
previously maintained.</em></p>
<p>Metasploit has already provide a nice write up of the pwning, I mean
testing the vector
<a href="">http://blog.metasploit.com/2010/08/exploiting-dll-hijacking-flaws.html</a>.
It does involve a bit of prep work but I tested it on a fully patched
Windows <span class="caps">XP</span> sp3 host and it does provide you with the same privileges as
the user who executes the exploit remotely giving the attacker access to
the system.</p>
<p>So we want to be concerned with how to prevent evil doers from
exploiting this vector.</p>
<p>\1. Do not open any network shares or websites that you are unfamiliar
with, furthermore avoid executing unknown files from either. 2. Decide
which workaround you would like to use per
<a href="http://www.microsoft.com/technet/security/advisory/2269637.mspx">http://www.microsoft.com/technet/security/advisory/2269637.mspx</a>.</p>
<ul>
<li>
<p>Workaround #1 Disabling and stopping the Webclient services is the
easiest method to prevent the attack but may cause other problems.</p>
</li>
<li>
<p>Workaround #2 Blocking ports 139 and 445 may not be ideal to block
due to file sharing and other problems that may arise.</p>
</li>
<li>
<p>Workaround #3 Download and install the tool from Microsoft that
allows control of the <span class="caps">DLL</span> search path algorithm from
<a href="http://support.microsoft.com/kb/2264107">http://support.microsoft.com/kb/2264107</a> for your specific
Microsoft distribution, i.e. Windows <span class="caps">XP</span>. Modify the registry key
that turns on, off or specifies the action per
<a href="http://support.microsoft.com/kb/2264107">http://support.microsoft.com/kb/2264107</a> section <strong>“Example 1:
How to disable loading DLLs from a WebDAV share for all applications
that are installed on your local computer”</strong>.</p>
</li>
</ul>
<p>Okay, so in short there are two ideal ways to disable to attack, disable
the Webclient service or install the tool and modify the specific
registry key.</p>
<ul>
<li>
<p>Note many of us run docked and undocked, therefore we need to modify
both controlset001 and controlset002 to cover both situations.</p>
<p>http://blog.metasploit.com/2010/08/exploiting-dll-hijacking-flaws.html</p>
</li>
</ul>Keeping your hardware safe and avoiding the evil maid2010-11-23T21:17:00-05:002010-11-23T21:17:00-05:00Stephen Reesetag:www.rsreese.com,2010-11-23:/keeping-your-hardware-safe-and-avoiding-the-evil-maid/This installment is about keeping your notebook and other technology items safe. I was recently asked what the Defcon locks were for that I have been distributing with the new notebooks. I jokingly said to keep people from taking your monitor and chair from your desk while your on travel …<p>This installment is about keeping your notebook and other technology
items safe. I was recently asked what the Defcon locks were for that I
have been distributing with the new notebooks. I jokingly said to keep
people from taking your monitor and chair from your desk while your on
travel but there is a better reason I distribute them.</p>
<p>People assume having your hardware stolen is the ultimate way to
compromise your data. An adversary that is smart enough will know better
though. A system running TrueCrypt or similar encryption is a near
impossible target if powered off while you are away but a system running
encryption that powered on on, not so much. Passwords and keys to most
encryption are <a href="http://en.wikipedia.org/wiki/TrueCrypt#Passwords_stored_in_memory">stored in memory</a> while the system is running.
Recovering said keys is not an easy task but is possible. If you cannot
break the habit of leaving your device on when not around or putting it
standby because you cannot stand the boot up time then make sure you are
using strong passwords that a difficult to guess to avoid giving the
attacker the chance to use tools to capture memory and parse it for your
super secret pass-phrase.</p>
<p>Even this has it’s downfalls though, there have been attacks that can
thwart the password mechanism on a device and run an attack such as
stealing the pass-phrase. An example is the <a href="http://www.hermann-uwe.de/blog/physical-memory-attacks-via-firewire-dma-part-1-overview-and-mitigation">Firewire attack</a> which
provides direct hardware access from some devices to your system. If the
attacker can do this then it is game over for your data as they can use
a tool to crack your system password. Fix, do not let an attacker walk
away with your device still powered on, i.e. use a lock when at clients
or at a hotel room.</p>
<p>The <a href="http://theinvisiblethings.blogspot.com/2009/10/evil-maid-goes-after-truecrypt.html">evil maid attack</a> is often not thought of. You are supporting a
remote client, come back to your room to check your mail and leave for
dinner leaving your notebook. While gone the evil-doer aka evil maid
visits your room to fluff your pillows and notices your notebook on the
table. Whether it’s on or off a device that you probably won’t notice is
plugged into your system and it records your pass-phrase when you type
it in. The evil maid returns to then steal the notebook as they now have
the passphrase to get your data. To avoid this one, pay attention to
rogue devices plugged into your hardware. Sounds simple but who would
check for a small <span class="caps">USB</span> device plugged into the back their host. Also use
a lock to keep the evil-doer from stealing the hardware after obtaining
the key after such an attack.</p>
<p>What am I trying to say here?</p>
<ul>
<li>Use encryption, the performance hit is very small and the newest
notebooks with the “i” series chipsets use hardware encryption.</li>
<li>Avoid leaving your device running if not around when at foreign
locations, i.e. hotels, clients, etc…</li>
<li>Use a lock to attach the notebook to a desk, chair, whatever. I know
these are not exactly Fort Knox but it is a deterrent.</li>
<li>Epoxy ports (warning this may not be an available option for a
corporate assets). Yes this is extreme but why do you think some
companies enforce this on their desktop systems and/or servers.</li>
</ul>Creating VMware VMDK files from DD images using Live View2010-11-07T01:18:00-05:002010-11-07T01:18:00-05:00Stephen Reesetag:www.rsreese.com,2010-11-07:/creating-vmware-vmdk-files-from-dd-images-using-live-view/While watching some Florida football today I decide to figure out how to mount/run a DD image in VMware Workstation. My image mounting skills were a little lacking so Google it was. I found a ton of great examples that seemed like they should work but the steps seemed …<p>While watching some Florida football today I decide to figure out how to
mount/run a <span class="caps">DD</span> image in <a href="http://http://www.vmware.com/products/workstation/">VMware Workstation</a>. My image mounting skills
were a little lacking so Google it was. I found a ton of great examples
that seemed like they should work but the steps seemed a little
incomplete. To further complicate the task was I was trying to run two
partitions from the same disk.</p>
<p>The first method I found was to manually create the <span class="caps">VMDK</span> file from
scratch. This seemed promising when I found
[http://sanbarrow.com/vmdk/disktypes.html#partitionedDevice][] and even
better an AppSpot application
<a href="http://www.schatzforensic.com.au/2006/p2v/">http://www.schatzforensic.com.au/2006/p2v/</a> to produce the
configuration for me but determining the <span class="caps">CHS</span> values were not going very
well with the images I was working with so I kept looking.</p>
<p><a href="http://www.techpathways.com/Demo.htm">ProDiscover</a> looked rather promising
<a href="https://irhowto.wordpress.com/2010/07/05/booting-a-dd-image-with-vmware/">https://irhowto.wordpress.com/2010/07/05/booting-a-dd-image-with-vmware/</a>
but the <span class="caps">VMDK</span> files generated for the images did not seem correct and
sure enough the guest system would not fire.</p>
<p><a href="http://liveview.sourceforge.net/">Live View</a> was the next tool to try. Initial attempts to use it on a
Windows 7 x64 host failed so I moved the image and required tools to a
Windows <span class="caps">XP</span> host. There are several prerequisites for Live View which it
will prompt you for so heads up.</p>
<ul>
<li>VMware Workstation or Server</li>
<li>Java or compatible <span class="caps">JRE</span></li>
<li><a href="https://www.vmware.com/support/developer/vddk/">Virtual Disk Development Kit (<span class="caps">VDDK</span>)</a></li>
<li><a href="http://liveview.sourceforge.net/">Live View</a></li>
</ul>
<p>You will need to create a <span class="caps">VMDK</span> for each image that you want to use even
if it is for the same <span class="caps">VM</span> guest. The coolest part of it all is that you
can use the <span class="caps">DD</span> image in a read-only state and all write are saved to a
separate state/snapshot file. Very nice as it keeps from trashing the
original image.</p>How I got started in information technology2010-08-17T14:29:00-04:002010-08-17T14:29:00-04:00Stephen Reesetag:www.rsreese.com,2010-08-17:/how-i-got-started-in-information-technology/Every once in a while someone asks me how I got started in working in the information technology realm. Usually someone that is not in the industry or they are interested in working with computers as a career and are not really sure where to start. I do not think …<p>Every once in a while someone asks me how I got started in working in
the information technology realm. Usually someone that is not in the
industry or they are interested in working with computers as a career
and are not really sure where to start. I do not think I have been able
to come up with a great answer but here is how it has worked for me thus far.</p>
<p>I have always had a mechanical inclination. I was one of those kids that
would rather take apart their toys (read break) then play with them. I
originally had a love affair with cars, especially engines. I would have
one of my parent’s take me to the junk yard (before I could drive) just
so I could pull old V8’s and bring them home to disassemble them. This
was entertaining but then our family got a new computer. I had worked
with friend’s computers but was careful not to break them as I knew
their cost. You can imagine my dad’s face when he brought home our first
computer and shortly thereafter I had the internals of it laid out
across the floor. Lucky for me I somehow was able to put it back
together and it still worked. I was hooked as there seemed like an
endless amount of possibilities to keep me occupied.</p>
<p>I continued on my quest of learning more by installing other operating
systems such as Redhat 6 besides the Windows 95 install as a dual boot
installation. Not a very interesting feat now but at the time it was
amazing for me. Fast forward a few years and I had gotten various jobs
working for firms setting up and maintaining computer systems. I
eventually got bit but the security bug while working at a university. I
find the aspect of securing computer systems quite interesting as not
only are you concerned with how information systems are implemented but
also what vectors may be used to attack them and so much more. Enough
about that here’s what I told the last person that was interested in
getting into the technology scene. Opinions vary greatly here.</p>
<p>It depends upon what you see yourself doing in 10, 20 years from now.
Computer Science (<span class="caps">CS</span>) degrees are great and they usually cover the
spectrum when it comes to the world of computing. I was going to get a
<span class="caps">CS</span> degree but was undecided the first two years and by the time I pulled
it together I realized I would need two years of Calculus and Physics
before most universities would even consider me for their programs. I
instead went the Computer Information Science (<span class="caps">CIS</span>) route. This worked
well for me as they are well recognized and the prerequisites were less
demanding and time consuming.</p>
<p>Many universities now offer a number of programs such as Decision
Information Science (<span class="caps">DIS</span>), this example focuses on more of the business
perspective. I know one person whom has gone this route but they have
done well. Most jobs will say they want a technology oriented degree
though are not always specific. Regardless do your research. This
ultimately depends upon what you expect to do and where you want to
work. If you know the type of position you might see yourself in then
look a position descriptions and figure out what the firms desire in
that field. There are plenty of jobs out there but just more competition
for them.</p>
<p>Due to competition in the market I would definitely recommend three
things. One, if feasible, regardless of the bachelors program get a
masters, these seems to open more doors and some schools have 3/2
programs that allow you to pretty much get a masters and bachelors at
almost the same time. Two, get an internship and/or job working with
computers, helpdesk at a university or work for a small company
maintaining their network, etc. Besides education, experience is highly
regarded in the industry regardless of your concentration and this will
help you figure out what you want to do career wise. Three, look into
certifications such as a <span class="caps">CCNA</span>, Security+, <span class="caps">MCSA</span>. Even entry level
certifications may help get you in the door though this is debatable by some.</p>
<p>I will state that I know people that rely purely upon their experience
and others that are more academically focused. I do not think there is a
sure fire method but for me a combination of both has worked fairly well.</p>Finally migrated from Blogger to WordPress2010-05-23T18:35:00-04:002010-05-23T18:35:00-04:00Stephen Reesetag:www.rsreese.com,2010-05-23:/finally-migrated-from-blogger-to-wordpress/I haven’t posted in a while because Blogger finally did away with their FTP/SCP publishing ability meaning if I wanted to continue using Google’s Blogger platform I would have to allow them to host my content for me. I don’t mind this except there are small …<p>I haven’t posted in a while because Blogger finally did away with their
<span class="caps">FTP</span>/<span class="caps">SCP</span> publishing ability meaning if I wanted to continue using
Google’s Blogger platform I would have to allow them to host my content
for me. I don’t mind this except there are small annoyances such as
having to still use a third party host for files that are not part of a
blog post. I have also never been a real fan of their themes. I’m not
much of a designer when it comes to websites, my focus is usually on the
technical operations and not making things aesthetically pleasing.
WordPress has Blogger beat hands down in this department as there are
thousands of freely available themes and plug-ins for their platform.</p>
<p>The flip-side is securing WordPress. There are countless known
vulnerabilities to the WordPress platform. There are ways to stay on top
of these. First use the general lock-down <a href="http://codex.wordpress.org/Hardening_WordPress" title="suggestions">suggestions</a> provided by
WordPress and other sites. Secondly or maybe primarily, stay up on new
releases that fix bugs and security vulnerabilities by subscribing to
the <a href="http://codex.wordpress.org/Mailing_Lists#Announcement_Mailing_Lists" title="mailing-list">mailing-list</a> or keeping an eye on their blog. Overall I look
forward to the new platform and hope you enjoy the content to come.</p>Redirect Blogger URL using Mod Rewrite and shell scripting fu2010-02-13T01:07:00-05:002010-02-13T01:07:00-05:00Stephen Reesetag:www.rsreese.com,2010-02-13:/redirect-blogger-url-using-mod-rewrite-and-shell-scripting-fu/Blogger is doing away with the option to host your blog via your own host and migrating everything to the cloud. I wanted to have the option to continue hosting my blog on my own server even though as of now I am still hosting with Blogger. The main concern …<p>Blogger is doing away with the <a href="http://blogger-ftp.blogspot.com/">option</a> to host your blog via your own
host and migrating everything to the cloud. I wanted to have the option
to continue hosting my blog on my own server even though as of now I am
still hosting with <a href="http://www.blogger.com/">Blogger</a>. The main concern I had was redirecting
URLs that blogger had created to a new blogging platform such as
<a href="http://www.wordpress.com/">WordPress</a>. I looked around and found several methods <a href="http://joepoon.com/blog/2009/04/02/from-blogger-to-wordpress-without-breaking-the-internet/">here</a>,
<a href="http://www.seobook.com/migrate-blogger-powered-blog-wordpress">here</a>, and <a href="http://www.slicksurface.com/blog/2008-03/how-to-migrate-from-blogger-to-wordpress">here</a> for redirecting one <span class="caps">URL</span> to another. The two
primary method were <span class="caps">HTTP</span> redirects by modifying the page header or
Apaches [mod_rewrite][]. I like Apache so I opted for the latter.</p>
<p>I only had about 60 posts so creating a few mod_rewrite rules is not a
big deal. There were a number bloggers had <a href="http://blogger-ftp.blogspot.com/2010/02/for-blogs-that-are-no-longer-updated.html">complaints</a> about Blogger
removing <span class="caps">FTP</span>/<span class="caps">SFTP</span> publishing capabilities and they were considering a
migration away from Blogger. This got me thinking about how to help
others in transferring thousands blog entries.</p>
<p>I decided to try to automate this process somewhat with a little
scripting fu. This could be scripted into a single script and if there
is enough interest, I will make it happen.</p>
<p>The first step is to import your Blogger posts into your WordPress
database. Blogger can export its posts but WordPress does not have a
native plug-in for importing the posts in the <span class="caps">XML</span> format that Blogger is
capable of exporting. WordPress can however import posts and comments
from a Blogger Blogspot hosted profile. Create a Blogspot host and
import the posts that you have backed up from your main profiles <span class="caps">XML</span>
file. Make sure to disable search engine indexing for the temporary site
so that you do not hurt your <span class="caps">SEO</span>.</p>
<p>The second step is to import the posts into WordPress. This is
relatively easy to do, basically login to your WordPress administrative
tools and import the blogger posts from your Blogspot profile that you
created in the first step. I tried using the recommended <a href="http://codex.wordpress.org/Importing_Content#Blogger">tools</a> per
WordPress and a third party <a href="http://justinsomnia.org/2006/10/maintain-permalinks-moving-from-blogger-to-wordpress/">tool</a> but they did not work very well for me.</p>
<p>Now your WordPress install should have all of your content and comments
and your WordPress install is working correctly. This tutorial also
assumes you are using the following permalink format for your WordPress
posts, if not you will have to adjust this tutorial to your liking:</p>
<div class="highlight"><pre><span></span>/%year%/%monthnum%/%postname%/
</pre></div>
<p>You will notice that your <span class="caps">URL</span> conforms to the WordPress install and not
to Bloggers. This means that when you migrate your <span class="caps">DNS</span> to point at your
shiny WordPress install all of the links that users have bookmarked and
the search engines have crawled will no longer be valid. Worse, this
could hurt your search engine rankings as it will take time for search
engines to realize the new content and during that time you will have
duplicate content floating around. Not an ideal situation.</p>
<p>Third step is to determine all of the URLs that your Blogger account was
using the <span class="caps">XML</span> file that you exported from your Blogger blogs profile.
This will produce a file with your Blogger file names. It should be the
same as the number of posts you have published on Blogger or in other
words imported to WordPress. Note you will need to change the <span class="caps">XML</span> file
name and domain name to match your settings:</p>
<div class="highlight"><pre><span></span># Produces blogger file names.
sed "s/\(href='[^']*'\)/\1\n/g" blog-02-04-2010.xml |
grep "href='http://www.rsreese.com/20.*html'" |
sed "s+.*href='http://www.domain.com/\(20[^']*\)'.*+\1+" |
sort -ut/ -k3 | xargs -I{} basename {} | sort -u > /tmp/blogger.txt
</pre></div>
<p>Next you want to generate a similar listing from your WordPress install
that is populated with all of your Blogger content. This involves
logging into your MySQL install and exporting a little data.</p>
<div class="highlight"><pre><span></span>mysql -u wordpress_user -p
mysql> USE wordpress_db;
mysql> SELECT post_name FROM wp_posts INTO OUTFILE '/tmp/wp.txt';
</pre></div>
<p>Next you want to ensure that your post line up from the two files. In my
case I had some that were not sorted exactly right, this basically let
me know how much manipulating I would have to do. Paste this into a file
on your Linux and provide executable permissions such as ‘chmod +x
filename’. Then run the file ‘/filename’. Note you will need to specify
the paths to your wp.txt and blogger.txt in the small script.</p>
<div class="highlight"><pre><span></span>paste blogger.txt wp.txt | while read Line
do set $Line
echo "This is from FileA: " $1
echo "This is from FileB: " $2
done
</pre></div>
<p>Lastly lets actually generate the mod_rewrite rules for Apache. Again
when this runs the sort function may not match up the file names exactly
right so you may have to do some manual manipulation.</p>
<div class="highlight"><pre><span></span>paste blogger.txt wp.txt | while read Line
do set $Line
echo 'RewriteRule ^([0-9]{4})/([0-9]{1,2})/'$1'$ $1/$2/'$2'/ [NC,R=301,L]'
done
</pre></div>
<p>You probably want to redirect the output to a file so you can go in and
fix the values that have not sorted correctly.</p>
<p>The last part of the configuration here is a section from my Apache
configuration file. I have also included a little bit to redirect the
feeds though for me this was not very important as I syndicate through
<a href="http://feedburner.google.com/">FeedBurner</a> allowing me to modify my feed without effect subscribers.</p>
<div class="highlight"><pre><span></span># This has two of my rewrite rules, I have many more but kept it brief for readability.
<span class="nt"><Directory</span> <span class="err">/var/www/apache2-default/wordpress</span><span class="nt">/></span>
RewriteEngine OnRewriteBase /wordpress/
RewriteRule ^atom.xml$ feed/ [NC,R=301,L]
RewriteRule ^rss.xml$ feed/ [NC,R=301,L]
RewriteRule ^([0-9]{4})/([0-9]{1,2})/adding-character-to-line-using-perl.html$ $1/$2/adding-a-character-to-a-line-using-perl/ [NC,R=301,L]
RewriteRule ^([0-9]{4})/([0-9]{1,2})/authenicating-kerberos-against-active.html$ $1/$2/authenicating-kerberos-against-active-directory/ [NC,R=301,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /wordpress/index.php [L]<span class="nt"></Directory></span>
</pre></div>
<p>Finally you should test your setup to determine that all of the links redirect.</p>
<div class="highlight"><pre><span></span>sed "s/\(href='[^']*'\)/\1\n/g" blog-02-07-2010.xml|
grep "href='http://www.rsreese.com/20.*html'" |
sed "s+.*href='\([^']*\)'.*+\1+" |
sort -ut/ -k3 > /tmp/full_blogger_urls.txt
</pre></div>
<p>Next you can use wget to test the URLs to make sure they all redirect correctly.</p>
<div class="highlight"><pre><span></span>wget -i /tmp/full_blogger_urls.txt
</pre></div>
<p>This tutorial is not an end all solution is not perfect by any means. It
still requires some manipulation of data but if you have a large number
of URLs to redirect then you may find it useful. Your mileage may vary
though if you have problems or recommendations than drop a comment…</p>A few tools that may help rid of malware2010-02-09T03:22:00-05:002010-02-09T03:22:00-05:00Stephen Reesetag:www.rsreese.com,2010-02-09:/a-few-tools-that-may-help-rid-of-malware/These tools may help rid a computer system of malware but be warned they can be very destructive to your system. In other words if you don’t know what you’re doing then backup what you can and take it to a professional. Ad-Aware - This seems to be a …<p>These tools may help rid a computer system of malware but be warned they
can be very destructive to your system. In other words if you don’t know
what you’re doing then backup what you can and take it to a professional.</p>
<ul>
<li><a href="http://www.lavasoft.com/products/ad_aware_free.php">Ad-Aware</a> - This seems to be a popular click and point tool</li>
<li><a href="http://www.safer-networking.org/en/download/index.html">Spybot - Search <span class="amp">&</span> Destroy</a> - Same as above</li>
<li><a href="http://technet.microsoft.com/en-us/sysinternals/bb897445.aspx">RootkitRevealer</a> - Older tool but still useful</li>
<li><a href="http://www.gmer.net/#files"><span class="caps">GMER</span></a> - Great manual tool but can cause more damage than good if
you do not know what you are doing.</li>
<li><a href="http://free.antivirus.com/hijackthis/">HijackThis</a> - Similar to above, if you do not know what to remove
manually then be careful as you could damage your system.</li>
<li><a href="http://vil.nai.com/vil/stinger/">McAfee Labs Stinger</a> - Detection tool from McAfee</li>
<li><a href="https://secure.sophos.com/products/free-tools/sophos-anti-rootkit/download/">Sophos Anti-Rootkit</a> - Requires sign-up to download, annoying to
say the least</li>
</ul>
<p>Of course keep your current anti-spyware and virus installs and
definitions up2date.</p>Setting up maildrop with Courier MTA2010-02-08T05:05:00-05:002010-02-08T05:05:00-05:00Stephen Reesetag:www.rsreese.com,2010-02-08:/setting-up-maildrop-with-courier-mta/Setting up maildrop with Courier MTA Before I get into the maildrop here are a few notes to myself for setting up Courier. Before running ./configure you should add ssl bin directory to your path To receive local mail indifferent of caps touch {your/etc/courier/dir}locallowercase Account postmaster …<p>Setting up maildrop with Courier <span class="caps">MTA</span></p>
<p>Before I get into the maildrop here are a few notes to myself for setting
up Courier.</p>
<p>Before running ./configure you should add ssl bin directory to your
path<br>
To receive local mail indifferent of caps touch {your/etc/courier/dir}locallowercase</p>
<p>Account postmaster@ <span class="caps">HAS</span> to be set up as well in the
/usr/lib/courier/etc/aliases/system file</p>
<p>To tell courier about hosted domains,</p>
<p>add domain to, /etc/courier/hosteddomains</p>
<p>then,as root, run makehosteddomains</p>
<p>and to tell courier to accept esmtp connections for the domain</p>
<p>add domains to /etc/courier/esmtpacceptmailfor.dir/domains</p>
<p>then,as root, run makeacceptmailfor</p>
<p>Also, the email account postmaster@ <span class="caps">HAS</span> to be set up as well.</p>
<p>Here is the maildrop stuff:</p>
<ol>
<li>
<p>Edit the “/usr/lib/courier/etc/maildroprc” to have “|
/usr/lib/courier/bin/maildrop” as your delivery method</p>
</li>
<li>
<p>Create a “$<span class="caps">HOME</span>/.mailfilter” file to be read by maildrop, there is
no need for the most part of a “.courier” since mail drop is already
being used!</p>
</li>
<li>
<p>Make sure your “/usr/lib/courier/etc/maildroprc” doesn’t kill the
install <span class="caps">IE</span>:</p>
</li>
</ol>
<div class="highlight"><pre><span></span>#attempt at a maildroprc file...
if ( $SIZE < 26144 )
{
exception {
xfilter "/usr/bin/spamassassin"
}
}
if (/\^X-Spam-Flag: \*YES/)
{
exception {
to "$HOME/Maildir/.Trash/"
}
}
\#else
\#{
\# exception {
\# to "$HOME/Maildir/"
\# }
\#}
</pre></div>
<p>The commented out part is no good since your “.mailfilter” will never be
read so <span class="caps">DON</span>’T specifiy the default delivery since no matter what unless
specified other wise by an exit command will courier deliver to the
default “$<span class="caps">HOME</span>/Maildir” also goes for the .mailfilter, no matter where u
send the mail to there is no need to send it to the default location
unless you have some crazy kaos going on that is beyond my lame howto =)</p>
<p>\4. The contents of your “.mailfilter should be something like the following:</p>
<p><span class="dquo">“</span>| /usr/lib/courier/bin/mailbot -t autoresponse -s ‘AutoGoAwayMessage’
-A ‘From: test@prcdigital.com’ /usr/sbin/sendmail -f “</p>
<p>A “autoresponse” file should be created and placed in the same $<span class="caps">HOME</span>
directory as the “.mailfilter” is located, though a universal file can
be created from multiple users to access if desired.</p>
<p>\5. “chmod 600 .mailfilter autoresponse”</p>
<p>Also the same user:group that is owner of the Maildir should also own
these two files so “chown user:group .mailfilter autoresponse”</p>
<p>or Once you get to maildrop, you don’t want to bounce it. Your best bet
is to just drop it. Also, I would suggest using spamc/spamd if at all
possible. This is what I would do:</p>
<div class="highlight"><pre><span></span> if ( $SIZE < 204800 ) { exception { xfilter "/usr/bin/spamc" } }
if ((/^X-Spam-Flag: YES/)) { if ((/^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*/)) { echo "***** Dropping 15+ Spam *****" EXITCODE = 0 exit } else { to "$HOME/Maildir/.Trash/" } } to "$HOME/Maildir/"
</pre></div>
<p>You can get rid of the echo if you don’t want an entry in the log when
it drops an email.</p>
<div class="highlight"><pre><span></span>if ((/^X-Spam-Flag: YES/))
</pre></div>
<p>Why double parentheses? This is what I am using and it is not working,
though it seemed to work until recently:</p>
<div class="highlight"><pre><span></span>if (/^X-Spam-Level: *\*\*\*\*\*\*\*/){ exception { to "/dev/null" }}
</pre></div>Migrating from Blogger to WordPress2010-02-04T01:06:00-05:002010-02-04T01:06:00-05:00Stephen Reesetag:www.rsreese.com,2010-02-04:/migrating-from-blogger-to-wordpress/Blogger is removing the functionality to host your own “Blogger” content by disabling the FTP/SFTP functionality from their system. I’m considering their hosting solution or migrating to a WordPress solution. If I stick with Google’s Blogger hosting then bandwidth should not ever be an issue as they …<p>Blogger is removing the functionality to host your own “Blogger” content
by <a href="http://blogger-ftp.blogspot.com/2010/01/deprecating-ftp.html">disabling</a> the <span class="caps">FTP</span>/<span class="caps">SFTP</span> functionality from their system. I’m
considering their hosting solution or migrating to a WordPress solution.</p>
<p>If I stick with Google’s Blogger hosting then bandwidth should not ever
be an issue as they have a distributed computing system. The only
downfall is that I’ll probably have to use a sub-domain to host any
static files. If I move to hosting my own WordPress then I’ll probably
have to increase my virtual host resources since <span class="caps">PHP</span> and MySQL will be
required therefore using more system resources. This also increases my
hosts vulnerability footprint. Not only am I essentially increasing
adding two services but WordPress has had its fair share of security issues.</p>
<p>If you want to stick with Blogger the simple alternative is just to
migrate to a hosted Blogspot and use <a href="http://www.google.com/support/blogger/bin/answer.py?hl=en&answer=55373">custom domains</a>. You can simply
point your <span class="caps">DNS</span> host domain.com or sub.domain.com to Google’s <span class="caps">DNS</span> servers
and within a short amount of time you will be up and running again. With
this said there are a number of variables that come into play.</p>
<p>Google’s Blogspot does not support subfolders, one alternative is to use
a <span class="caps">URL</span> redirection to point to the new host which means you will need to
search around for the code to insert into the header of your template to
accomplish this. Per the <a href="http://blogger-ftp.blogspot.com/2010/01/migration-tool-overview.html">migration tool</a> there is no sub-folder support.</p>
<blockquote>
<p>domain.com/blog/ —> blog.domain.com</p>
</blockquote>
<p>Since Google would hosting your blog there really isn’t a wonderful way
to handle this as there is not a provision to use Mod_Rewrite or
something similar though with the number of complaints Google has
received on their blog they may implement a feature.</p>
<p>If you are considering hosting with another solution such as Wordpress
then you have more options available to you depending on your hosting
solution. Wordpress has an integrated import function to import other
Blogging but you must first convert you existing hosted Blogger account
to a Blogspot solution. Blogger does have an export function but it
seems broken per these <a href="http://blogger-ftp.blogspot.com/2010/02/for-blogs-that-are-no-longer-updated.html">posts</a>. Wordpress also has custom <span class="caps">URL</span>
functionality so it would be easier to match the format that blogger was
using especially if you can utilize Mod_Rewrite.</p>
<p>Personally, I’m still undecided…</p>God Mode - Give Windows users an easier way to destory their computers2010-01-06T03:45:00-05:002010-01-06T03:45:00-05:00Stephen Reesetag:www.rsreese.com,2010-01-06:/god-mode-give-windows-users-an-easier-way-to-destory-their-computers/Windows 7 and Vista (latter can be buggy) has an interesting feature that allows quick access to allow kinds of administrative tools. To create God Mode simply create a new folder on your desktop and name it the following: GodMode.{ED7BA470-8E54-465E-825C-99712043E01C} Now you will have a quicker way to change …<p>Windows 7 and Vista (latter can be buggy) has an interesting feature that allows quick access to allow kinds of administrative tools.</p>
<p>To create God Mode simply create a new folder on your desktop and name it the following:</p>
<div class="highlight"><pre><span></span>GodMode.{ED7BA470-8E54-465E-825C-99712043E01C}
</pre></div>
<p>Now you will have a quicker way to change settings that will probably lead to the demise of your operating system. Have fun.</p>Google namebench helps find happy nameservers2009-12-15T03:22:00-05:002009-12-15T03:22:00-05:00Stephen Reesetag:www.rsreese.com,2009-12-15:/google-namebench-helps-find-happy-nameservers/I was recently checking name servers that I was using to resolve hosts on a network. After using tools such as ping, traceroute, and dig I decided to search around and found Google has a new tool called namebench. Intrigued I decided to give it a shot. There is support …<p>I was recently checking name servers that I was using to resolve hosts
on a network. After using tools such as ping, traceroute, and dig I
decided to search around and found Google has a new tool called
<a href="http://code.google.com/p/namebench/">namebench</a>. Intrigued I decided to give it a shot. There is support
for several platforms including Linux and Microsoft Windows. I pulled
down a <span class="caps">NIX</span> copy and fired up the python script. By default in <a href="http://code.google.com/p/namebench/w/list"><span class="caps">CLI</span></a>
the tool tests the top 10000 Alexa sites, as a note the <a href="http://code.google.com/p/namebench/w/list"><span class="caps">GUI</span></a> tool
can test sites from your browsers cache. The tool compares your <span class="caps">DNS</span>
hosts to several top resolvers around the net including their <a href="http://code.google.com/speed/public-dns/">own</a>.
This was neat but I found the real usefulness was the ability to only
specify the name servers you want to test. Very cool <span class="caps">IMO</span>.</p>
<div class="highlight"><pre><span></span>$ ./namebench.py -O <span class="m">68</span>.87.73.242 <span class="m">68</span>.87.68.162 <span class="m">68</span>.87.74.162 <span class="m">8</span>.8.8.8 <span class="m">8</span>.8.4.4 <span class="m">208</span>.67.220.220
namebench <span class="m">1</span>.0.5 - data/alexa-top-10000-global.txt <span class="o">(</span>weighted<span class="o">)</span> on <span class="m">2009</span>-12-14 <span class="m">22</span>:09:40.248541
<span class="nv">threads</span><span class="o">=</span><span class="m">40</span> <span class="nv">tests</span><span class="o">=</span><span class="m">200</span> <span class="nv">runs</span><span class="o">=</span><span class="m">1</span> <span class="nv">timeout</span><span class="o">=</span><span class="m">2</span>.0 <span class="nv">health_timeout</span><span class="o">=</span><span class="m">4</span>.0 <span class="nv">servers</span><span class="o">=</span><span class="m">10</span>
------------------------------------------------------------------------------
- Checking connection quality...
- Connection appears healthy <span class="o">(</span>latency <span class="m">55</span>.15ms<span class="o">)</span>
- Building initial DNS cache <span class="k">for</span> <span class="m">6</span> nameservers <span class="o">[</span><span class="m">40</span> threads<span class="o">]</span>
- Waiting <span class="k">for</span> health check threads <span class="k">for</span> <span class="m">6</span> servers: <span class="m">0</span>/6.6/6
.- <span class="m">6</span> of <span class="m">6</span> name servers are healthy
- Waiting <span class="k">for</span> wildcard check threads: <span class="m">1</span>/6.....6/6
.- Waiting 4s <span class="k">for</span> TTLs to decrement.
- Waiting <span class="k">for</span> cache collusion threads: <span class="m">0</span>/30.30/30
<span class="m">30</span>
Final list of nameservers considered:
------------------------------------------------------------------------------
<span class="m">68</span>.87.68.162 <span class="m">68</span>.87.68.162 <span class="m">48</span> ms <span class="p">|</span>
<span class="m">208</span>.67.220.220 <span class="m">208</span>.67.220.220 <span class="m">59</span> ms <span class="p">|</span> www.google.com. hijacked <span class="o">(</span>google.navigation.opendns.com.<span class="o">)</span>, NXDOMAIN Hijacking
<span class="m">68</span>.87.73.242 <span class="m">68</span>.87.73.242 <span class="m">62</span> ms <span class="p">|</span>
<span class="m">68</span>.87.74.162 <span class="m">68</span>.87.74.162 <span class="m">78</span> ms <span class="p">|</span>
<span class="m">8</span>.8.8.8 <span class="m">8</span>.8.8.8 <span class="m">86</span> ms <span class="p">|</span>
<span class="m">8</span>.8.4.4 <span class="m">8</span>.8.4.4 <span class="m">88</span> ms <span class="p">|</span>
- Reading <span class="nb">test</span> data from data/alexa-top-10000-global.txt
- Benchmarking <span class="m">6</span> server<span class="o">(</span>s<span class="o">)</span>, run <span class="m">1</span> of <span class="m">1</span>: <span class="m">1</span>/200.........10.........20.........30.........40.........50.........60.........70.........80.........90.........100.........110.........120.........130.........140.........150.........160.........170.........180.........190.........200/200
<span class="m">200</span>
- Rendering template: ascii.tmpl
- Saving rendered ascii output
Fastest individual response <span class="o">(</span>in milliseconds<span class="o">)</span>:
----------------------------------------------
<span class="m">68</span>.87.68.162 <span class="c1">############################ 32.37295</span>
<span class="m">68</span>.87.73.242 <span class="c1">################################# 38.33604</span>
<span class="m">208</span>.67.220.220 <span class="c1">################################# 39.38794</span>
<span class="m">68</span>.87.74.162 <span class="c1">########################################## 49.34692</span>
<span class="m">8</span>.8.4.4 <span class="c1">##################################################### 63.43389</span>
<span class="m">8</span>.8.8.8 <span class="c1">##################################################### 63.49301</span>
Mean response <span class="o">(</span>in milliseconds<span class="o">)</span>:
--------------------------------
<span class="m">8</span>.8.4.4 <span class="c1">########################## 67.35</span>
<span class="m">68</span>.87.73.242 <span class="c1">################################## 90.54</span>
<span class="m">8</span>.8.8.8 <span class="c1">#################################### 95.24</span>
<span class="m">68</span>.87.68.162 <span class="c1">#################################### 95.31</span>
<span class="m">208</span>.67.220.220 <span class="c1">######################################### 108.85</span>
<span class="m">68</span>.87.74.162 <span class="c1">##################################################### 142.74</span>
Response Distribution Chart URL <span class="o">(</span>200ms<span class="o">)</span>:
----------------------------------------
http://chart.apis.google.com/chart?cht<span class="o">=</span>lxy<span class="p">&</span><span class="nv">chs</span><span class="o">=</span>720x410<span class="p">&</span><span class="nv">chxt</span><span class="o">=</span>x,y<span class="p">&</span><span class="nv">chg</span><span class="o">=</span><span class="m">10</span>,20<span class="p">&</span><span class="nv">chxr</span><span class="o">=</span><span class="m">0</span>,0,200<span class="p">|</span><span class="m">1</span>,0,100<span class="p">&</span><span class="nv">chd</span><span class="o">=</span>t:0,20,20,20,21,21,21,24,27,49,59,67,116<span class="p">|</span><span class="m">0</span>,1,12,40,57,63,69,73,77,80,84,87,91<span class="p">|</span><span class="m">0</span>,16,17,17,18,19,25,26,29,35,39,51,77,95,102<span class="p">|</span><span class="m">0</span>,1,14,28,48,53,56,60,65,69,72,76,80,83,87<span class="p">|</span><span class="m">0</span>,19,20,20,20,21,21,23,24,26,36,56,69,90,112<span class="p">|</span><span class="m">0</span>,1,8,30,45,50,54,61,64,70,73,77,80,84,87<span class="p">|</span><span class="m">0</span>,25,25,25,26,27,45,46,49,51,57,71,77,91,116<span class="p">|</span><span class="m">0</span>,1,5,33,39,47,50,54,58,62,65,69,73,77,80<span class="p">|</span><span class="m">0</span>,32,32,32,33,33,34,34,35,38,48,53<span class="p">|</span><span class="m">0</span>,1,7,28,55,65,80,88,91,95,98,100<span class="p">|</span><span class="m">0</span>,32,32,32,33,33,34,34,34,37,41,50,63,78,126<span class="p">|</span><span class="m">0</span>,1,7,28,44,54,66,70,74,77,81,85,89,92,96<span class="p">&</span><span class="nv">chco</span><span class="o">=</span>ff9900,1a00ff,80ff00,ff00e6,00e6ff,fae30a<span class="p">&</span><span class="nv">chxt</span><span class="o">=</span>x,y,x,y<span class="p">&</span><span class="nv">chxl</span><span class="o">=</span><span class="m">2</span>:<span class="o">||</span>Duration+in+ms<span class="o">||</span><span class="m">3</span>:<span class="o">||</span>%25<span class="p">|&</span><span class="nv">chdl</span><span class="o">=</span><span class="m">208</span>.67.220.220<span class="p">|</span><span class="m">68</span>.87.68.162<span class="p">|</span><span class="m">68</span>.87.73.242<span class="p">|</span><span class="m">68</span>.87.74.162<span class="p">|</span><span class="m">8</span>.8.4.4<span class="p">|</span><span class="m">8</span>.8.8.8
Response Distribution Chart URL <span class="o">(</span>Full<span class="o">)</span>:
---------------------------------------
http://chart.apis.google.com/chart?cht<span class="o">=</span>lxy<span class="p">&</span><span class="nv">chs</span><span class="o">=</span>720x410<span class="p">&</span><span class="nv">chxt</span><span class="o">=</span>x,y<span class="p">&</span><span class="nv">chg</span><span class="o">=</span><span class="m">10</span>,20<span class="p">&</span><span class="nv">chxr</span><span class="o">=</span><span class="m">0</span>,0,1333<span class="p">|</span><span class="m">1</span>,0,100<span class="p">&</span><span class="nv">chd</span><span class="o">=</span>t:0,3,3,3,3,3,3,4,4,7,9,10,17,23,62,100<span class="p">|</span><span class="m">0</span>,1,12,40,57,63,69,73,77,80,84,87,91,94,98,100<span class="p">|</span><span class="m">0</span>,2,2,3,3,3,4,4,4,5,6,8,12,14,15,19,22,24,60<span class="p">|</span><span class="m">0</span>,1,14,28,48,53,56,60,65,69,72,76,80,83,87,90,94,97,100<span class="p">|</span><span class="m">0</span>,3,3,3,3,3,3,3,4,4,5,8,10,13,17,20,23,25,32<span class="p">|</span><span class="m">0</span>,1,8,30,45,50,54,61,64,70,73,77,80,84,87,91,94,98,100<span class="p">|</span><span class="m">0</span>,4,4,4,4,4,7,7,7,8,9,11,11,14,17,19,22,25,28,45,67<span class="p">|</span><span class="m">0</span>,1,5,33,39,47,50,54,58,62,65,69,73,77,80,84,88,91,95,98,100<span class="p">|</span><span class="m">0</span>,5,5,5,5,5,5,5,5,6,7,8<span class="p">|</span><span class="m">0</span>,1,7,28,55,65,80,88,91,95,98,100<span class="p">|</span><span class="m">0</span>,5,5,5,5,5,5,5,5,6,6,8,9,12,19,55,69<span class="p">|</span><span class="m">0</span>,1,7,28,44,54,66,70,74,77,81,85,89,92,96,99,100<span class="p">&</span><span class="nv">chco</span><span class="o">=</span>ff9900,1a00ff,80ff00,ff00e6,00e6ff,fae30a<span class="p">&</span><span class="nv">chxt</span><span class="o">=</span>x,y,x,y<span class="p">&</span><span class="nv">chxl</span><span class="o">=</span><span class="m">2</span>:<span class="o">||</span>Duration+in+ms<span class="o">||</span><span class="m">3</span>:<span class="o">||</span>%25<span class="p">|&</span><span class="nv">chdl</span><span class="o">=</span><span class="m">208</span>.67.220.220<span class="p">|</span><span class="m">68</span>.87.68.162<span class="p">|</span><span class="m">68</span>.87.73.242<span class="p">|</span><span class="m">68</span>.87.74.162<span class="p">|</span><span class="m">8</span>.8.4.4<span class="p">|</span><span class="m">8</span>.8.8.8
Recommended configuration <span class="o">(</span>fastest + nearest<span class="o">)</span>:
----------------------------------------------
nameserver <span class="m">8</span>.8.4.4 <span class="c1"># 8.8.4.4</span>
nameserver <span class="m">68</span>.87.68.162 <span class="c1"># 68.87.68.162</span>
nameserver <span class="m">68</span>.87.73.242 <span class="c1"># 68.87.73.242</span>
</pre></div>Problem with RAID volume larger then 2TB on Dell workstations2009-10-21T14:35:00-04:002009-10-21T14:35:00-04:00Stephen Reesetag:www.rsreese.com,2009-10-21:/problem-with-raid-volume-larger-then-2tb-on-dell-workstations/I ran into a interesting issue this weekend. I was setting up a RAID volume on a Optiplex and Precision workstations, which have three 1.5 Terabyte (TB) drives. I tried creating a single large RAID 5 volume but the Intel Matrix storage manger (8.5.2) would not set …<p>I ran into a interesting issue this weekend. I was setting up a <span class="caps">RAID</span>
volume on a Optiplex and Precision workstations, which have three 1.5
Terabyte (<span class="caps">TB</span>) drives. I tried creating a single large <span class="caps">RAID</span> 5 volume but
the Intel Matrix storage manger (8.5.2) would not set the array to
bootable. After much trial I found I could create smaller volume 160
Gigabyte (<span class="caps">GB</span>) for the system which was bootable and another utilizing
the rest of the storage. My original plan was to create a large volume
and partition it using the <span class="caps">OS</span> but this worked just as well, so instead I
had two <span class="caps">RAID</span> 5 volumes. The only difference is the large volume is not
bootable and requires the small one with the <span class="caps">OS</span> on it to first be mounted.</p>Python File Uploader2009-10-17T14:21:00-04:002009-10-17T14:21:00-04:00Stephen Reesetag:www.rsreese.com,2009-10-17:/python-file-uploader/I recently had a need to upload large files to a server via HTTP. Most of the solutions required tweaking the web server or PHP. Instead, I found a Python script that would write the data in chunks so it could handle large files. I modified the script to include …<p>I recently had a need to upload large files to a server via <span class="caps">HTTP</span>. Most of the solutions required tweaking the web server or <span class="caps">PHP</span>. Instead, I found a <a href="http://python.org/">Python</a> script that would write the data in chunks so it could handle large files. I modified the script to include a few additional features which include reporting a hash to the user, appending a date and revision to the file. I did my testing with <a href="http://apache.org/">Apache</a> so your mileage may very with other httpd instances. The script is released under to GNUv3 so feel free to download a copy for your use or destruction. You can find the script at <a href="https://github.com/rsreese/file-uploader">Github</a>.</p>Trouble accessing Gmail or internal chat client2009-09-10T01:58:00-04:002009-09-10T01:58:00-04:00Stephen Reesetag:www.rsreese.com,2009-09-10:/trouble-accessing-gmail-or-internal-chat-client/I have been in a couple of places which I needed to access my email and chat so here is a little fix to get around DNS fixes that redirect hosts to the localhost. Modify your hosts file to look like the following: C:\Windows\System32\drivers\etc\hosts 127 …<p>I have been in a couple of places which I needed to access my email and chat so here is a little fix to get around <span class="caps">DNS</span> fixes that redirect hosts to the localhost.</p>
<p>Modify your hosts file to look like the following:</p>
<p>C:\Windows\System32\drivers\etc\hosts</p>
<div class="highlight"><pre><span></span>127.0.0.1 localhost
74.125.79.17 mail.google.com
66.102.1.189 chatenabled.mail.google.com #or 74.125.19.189 for non-Comcast.
# A friend recommended these, YMMV.
#209.85.135.17 mail.google.com
#72.14.204.189 chatenabled.mail.google.com
#72.14.204.189 talk.google.com
#72.14.204.189 talkx.l.google.com
#72.14.204.189 hostedtalkgadget.google.com
#72.14.204.189 talkgadget.google.com
</pre></div>
<p>This basically tell your system where these services are located instead of relying on a third party to instead let you know.</p>Facebook gets linked account support2009-05-20T03:04:00-04:002009-05-20T03:04:00-04:00Stephen Reesetag:www.rsreese.com,2009-05-20:/facebook-gets-linked-account-support/Now you can logon to your Facebook account through several providers such as Google, Myspace and OpenId which IMO is great (I’m lazy). Just go to Settings, Account Settings and Linked Accounts. You can even pick multiple providers. One cool part is my openID provider VeriSign can be setup …<p>Now you can logon to your Facebook account through several providers
such as Google, Myspace and OpenId which <span class="caps">IMO</span> is great (I’m lazy). Just
go to Settings, Account Settings and Linked Accounts. You can even pick
multiple providers. One cool part is my openID provider VeriSign can be
setup to use two factor authentication to help provide a little more
security amongst all of the chaos. See <a href="https://pip.verisignlabs.com/">https://pip.verisignlabs.com</a></p>
<p>Update 1 - As of now Google as a Linked Account is not logging me in
though <span class="lgtxtBl">pip.verisignlabs.com is still working well.</p>
<p>Update 2 - My Google account will log me into FaceBook once I have
authenticated via Gmail.<br>
</span></p>Installing Sun Java on Debian Lenny2009-05-15T15:04:00-04:002009-05-15T15:04:00-04:00Stephen Reesetag:www.rsreese.com,2009-05-15:/installing-sun-java-on-debian-lenny/The Sun Java JDK is available in the Debian Lenny non-free repository, therefore you must modify /etc/apt/sources.list: $ sudo vi /etc/apt/sources.list Add non-free to the Debian Lenny repositories: deb http://mirrors.kernel.org/debian/ lenny main non-freedeb-src http://mirrors.kernel.org/debian/ lenny main non-free …<p>The Sun Java <span class="caps">JDK</span> is available in the Debian Lenny non-free repository, therefore you must modify /etc/apt/sources.list:</p>
<div class="highlight"><pre><span></span>$ sudo vi /etc/apt/sources.list
</pre></div>
<p>Add non-free to the Debian Lenny repositories:</p>
<div class="highlight"><pre><span></span><span class="k">deb</span> <span class="s">http://mirrors.kernel.org/debian/</span> <span class="kp">lenny</span> <span class="kp">main</span> <span class="kp">non-freedeb-src</span> <span class="kp">http://mirrors.kernel.org/debian/</span> <span class="kp">lenny</span> <span class="kp">main</span> <span class="kp">non-free</span>
<span class="k">deb</span> <span class="s">http://security.debian.org/</span> <span class="kp">lenny/updates</span> <span class="kp">main</span> <span class="kp">non-freedeb-src</span> <span class="kp">http://security.debian.org/</span> <span class="kp">lenny/updates</span> <span class="kp">main</span> <span class="kp">non-free</span>
</pre></div>
<p>Run</p>
<div class="highlight"><pre><span></span>$ sudo apt-get update
</pre></div>
<p>Install the Java <span class="caps">JDK</span> as follows:</p>
<div class="highlight"><pre><span></span>$ sudo apt-get install sun-java6-jdk
</pre></div>
<p>Make it available system wide:</p>
<div class="highlight"><pre><span></span>$ sudo update-java-alternatives -s java-6-sun <span class="nb">echo</span> <span class="s1">'JAVA_HOME="/usr/lib/jvm/java-6-sun"'</span> <span class="p">|</span> tee -a /etc/environment
</pre></div>Debian Backup Script2009-03-02T01:30:00-05:002009-03-02T01:30:00-05:00Stephen Reesetag:www.rsreese.com,2009-03-02:/debian-backup-script/The script is located here. It can update the software repository, backup the file system, and send the backup to another machine via SSH. Feel free to try it out and let me know if you have any issues. Shell script to update Debian system via APT. Backup systems and …<p>The script is located <a href="https://github.com/rsreese/debian-update-script">here</a>. It can update the software repository, backup the file system, and send the backup to another machine via <span class="caps">SSH</span>. Feel free to try it out and let me know if you have any issues. </p>
<ul>
<li>Shell script to update Debian system via <span class="caps">APT</span>. </li>
<li>Backup systems and send the backups to remote systems </li>
<li>MySQL backup </li>
<li>Encrypted backups available </li>
<li>System information like disc usage, network traffic </li>
<li>Log file output from syslog</li>
</ul>A few simple computing tips2009-02-10T01:46:00-05:002009-02-10T01:46:00-05:00Stephen Reesetag:www.rsreese.com,2009-02-10:/a-few-simple-computing-tips/Here’s a short list of safe computing tips that may help you stay safe. \1. Passwords, use complex passwords and do not use the same password for MySpace/Facebook as you do for your banking website. This is an easy habit to get into so try to break the …<p>Here’s a short list of safe computing tips that may help you stay safe.</p>
<p>\1. Passwords, use complex passwords and do not use the same password
for MySpace/Facebook as you do for your banking website. This is an easy
habit to get into so try to break the mold and use something complex
that uses numbers, letters, and special characters.</p>
<p>\2. Encryption, this is a must for notebooks and other portable devices.
Most individuals do not think about it until the worst happens but how
bad would it suck to have your notebook or whatever stolen and then the
thief happen to be intelligent enough to data mine through you drive to
find credit card numbers or whatever other goodies they could use to
steal your identity. There are some good free encryption software
packages out there so do a little research.</p>
<p>\3. Avoid intercepted data. Most people do not think about how the data
gets from their web browser to it’s destination but I can tell you a
majority of the time your data that is trans-versing networks that you
have no control over is probably unencrypted therefore not secure and up
for being intercepted. Pay attention to what you say over instant
messaging and other forms of communication as you would be very
surprised as to whom might be listening and worse capturing your information.</p>
<p>\4. Backups, the medium in which your data resides on more then likely
has a shelf life so a little thought in regards to backing up your data
can go a long way if data becomes corrupt, drive failure, or by
malicious means.</p>New RSS feed2009-01-23T04:19:00-05:002009-01-23T04:19:00-05:00Stephen Reesetag:www.rsreese.com,2009-01-23:/new-rss-feed/Tinkering as usual I was check out my FeedBurner feeds for accuracy since I have heard through the grapevine that a number of users are having problems with incorrect feed statistics when using FeedBurner. My statistics seem to be fine (not like anyone subscribes anyhow :-). It was interesting that Google …<p>Tinkering as usual I was check out my <a href="http://www.feedburner.com/fb/a/home">FeedBurner</a> feeds for accuracy
since I have heard through the grapevine that a number of users are
having problems with incorrect feed statistics when using FeedBurner. My
statistics seem to be fine (not like anyone subscribes anyhow :-). It
was interesting that Google has acquired FeedBurner and are planning on
migrating the <span class="caps">FB</span> user base to Google though I have yet to receive any
notification which was disappointing… The migration was painless
enough and if you feel inclined my new feed is available at:
<a href="http://feedproxy.google.com/rsreese">http://feedproxy.google.com/rsreese</a>.</p>TrueCrypt on my Dell notebook2008-12-19T00:23:00-05:002008-12-19T00:23:00-05:00Stephen Reesetag:www.rsreese.com,2008-12-19:/truecrypt-on-my-dell-notebook/So I recently acquired a new notebook and I of course wanted the notebook to be secure. When I say secure I’m not just talking about preventing someone from exploiting the notebook from the wild but the problem of physical security with regards to someone stealing it. There are …<p>So I recently acquired a new notebook and I of course wanted the
notebook to be secure. When I say secure I’m not just talking about
preventing someone from exploiting the notebook from the wild but the
problem of physical security with regards to someone stealing it. There
are a number of commercial tools out there to provide whole disk
encryption (<span class="caps">WDE</span>) but I really did not want to spend the money so I
decided to get <a href="http://www.truecrypt.org/docs/?s=system-encryption">TrueCrypt</a> a shot. I’ve been using it for sometime to
encrypt data on a few backup drives I have but never a system drive. The
<a href="http://www.truecrypt.org/docs/?s=system-encryption">process</a> is completely painless. I decided to stick with the
<a href="http://www.truecrypt.org/docs/?s=aes"><span class="caps">AES</span></a> algorithm since it’s less hardware intense but be aware there
are stronger encryption schemes available from the product. I also
recommend making a backup disk and testing it! Secondly do <span class="caps">NOT</span> lose your
key or you will not get into the system so it may be ideal to make
backups and place them on another medium just in case…</p>
<p>At this point I’m rather happy with TrueCrypt the performance is great
and how cool is it having the piece of mind that if someone decides to
take your hardware, it is currently impossible for them to retrieve your data.</p>Using session-monitor to span ports as an aggregation tap2008-10-17T20:13:00-04:002008-10-17T20:13:00-04:00Stephen Reesetag:www.rsreese.com,2008-10-17:/using-session-monitor-to-span-ports-as-an-aggregation-tap/Like most I do not have the funds to purchase a $1000 port aggregation tap for my IDS to monitor traffic so instead I just used a 2950 Cisco Switch: ! interface FastEthernet0/1 switchport access vlan 100 duplex full ! interface FastEthernet0/2 switchport access vlan 100 duplex full ! interface FastEthernet0 …<p>Like most I do not have the funds to purchase a $1000 port aggregation
tap for my <span class="caps">IDS</span> to monitor traffic so instead I just used a 2950 Cisco Switch:</p>
<div class="highlight"><pre><span></span><span class="o">!</span>
<span class="kr">interface</span> <span class="nx">FastEthernet0</span><span class="o">/</span><span class="mi">1</span>
<span class="nx">switchport</span> <span class="nx">access</span> <span class="nx">vlan</span> <span class="mi">100</span>
<span class="nx">duplex</span> <span class="nx">full</span>
<span class="o">!</span>
<span class="kr">interface</span> <span class="nx">FastEthernet0</span><span class="o">/</span><span class="mi">2</span>
<span class="nx">switchport</span> <span class="nx">access</span> <span class="nx">vlan</span> <span class="mi">100</span>
<span class="nx">duplex</span> <span class="nx">full</span>
<span class="o">!</span>
<span class="kr">interface</span> <span class="nx">FastEthernet0</span><span class="o">/</span><span class="mi">3</span>
<span class="o">!</span>
</pre></div>
<p>so the first two ports are where the traffic comes in and back out to
the destination device, the third will go to my network sensor. Next
let us setup the port spanning.</p>
<div class="highlight"><pre><span></span>!
monitor session 1 source interface Fa0/1
monitor session 1 destination interface Fa0/3
</pre></div>
<p>Note that you may check other options such as spanning multiple ports or
even vlans.</p>Using metasploit to pwn MS06-0672008-10-10T00:02:00-04:002008-10-10T00:02:00-04:00Stephen Reesetag:www.rsreese.com,2008-10-10:/using-metasploit-to-pwn-ms06-067/In a graduate course I was taking, our professor wanted us to tool around with the Metasploit project. This tool makes quick work of exploiting vulnerabilities. After the client takes the opens the link, I ran ‘ipconfig’ to ensure I had remote connectivity. Here a shell that I ran ‘ipconfig …<p>In a graduate course I was taking, our professor wanted us to tool around with the <a href="http://www.metasploit.com/">Metasploit</a> project. This tool makes quick work of exploiting vulnerabilities. After the client takes the opens the link, I ran ‘ipconfig’ to ensure I had remote connectivity.</p>
<p><img alt="image1" src="https://www.rsreese.com/assets/SPVI4KP-725225.PNG"></p>
<p>Here a shell that I ran ‘ipconfig’ on just to confirm the operation.</p>
<p><img alt="image3" src="https://www.rsreese.com/assets/SPVI4KO-777718.PNG"></p>Erase slack space on Microsoft Vista2008-10-03T04:34:00-04:002008-10-03T04:34:00-04:00Stephen Reesetag:www.rsreese.com,2008-10-03:/erase-slack-space-on-microsoft-vista/A lot of information may be stored on a drives slack space. If you want to get rid of these artifacts then run the usual tools to clean up the system like ‘Disk Cleanup’, ‘Defrag’, etc.. and then run the following command. C:\Users\Crypto>cipher.exe /w:C: To …<p>A lot of information may be stored on a drives <a href="http://en.wikipedia.org/wiki/Fragmentation_%28computer%29">slack space</a>. If you
want to get rid of these artifacts then run the usual tools to clean up
the system like ‘Disk Cleanup’, ‘Defrag’, etc.. and then run the
following command.</p>
<div class="highlight"><pre><span></span>C:\Users\Crypto>cipher.exe /w:C:
To remove as much data as possible, please close all other applications while
running CIPHER /W.
Writing 0x00...................................................................................................
Writing 0xFF...................................................................................................
Writing Random Numbers.........................................................................................
</pre></div>Gentoo Linux auto update script2008-09-08T04:14:00-04:002008-09-08T04:14:00-04:00Stephen Reesetag:www.rsreese.com,2008-09-08:/gentoo-linux-auto-update-script/A script that I had been using for sometime to update my Gentoo servers needed a few additions in my opinion. I spoke to the original developer of the script and he allowed me to make additions to the script and post them here on Google’s code hosting server …<p>A script that I had been using for sometime to update my Gentoo servers
needed a few additions in my opinion. I spoke to the <a href="http://monkey-house-org.blogspot.com/2007/06/gentoo-auto-update-scripts.html">original
developer</a> of the script and he allowed me to make additions to the
script and post them <a href="http://code.google.com/p/gentoo-update-script/">here</a> on Google’s code hosting server. The
following is a basic description of the script. So if you’re looking for
something to update your Gentoo boxes then cruise over and pickup a copy.</p>
<p><span class="dquo">“</span>Shell script for Gentoo Linux to preform nightly system administration
tasks from a cron job. This is reminiscent of OpenBSD’s /etc/daily,
weekly, monthly scripts. Includes auto updating for Nikto, Snort sigs,
and Nessus plugins. Also includes MySQL dump support, file system
backups, and remote backups via <span class="caps">SSH</span>/rysnc.”</p>Mounting drives/volumes read-only in Microsoft Windows (Vista)2008-08-05T21:35:00-04:002008-08-05T21:35:00-04:00Stephen Reesetag:www.rsreese.com,2008-08-05:/mounting-drivesvolumes-read-only-in-microsoft-windows-vista/I needed to analyze a drive for a company that suspects an ex-employee may have taken corporate material (training exercise or else I would use a hardware write blocker and follow a chain of custody). I do not have a write blocker and rather then fire up a copy of …<p>I needed to analyze a drive for a company that suspects an ex-employee
may have taken corporate material (training exercise or else I would use
a hardware write blocker and follow a chain of custody). I do not have a
write blocker and rather then fire up a copy of Helix or a similar tool
a my spare machine (which is painfully slow) I would rather perform
analysis on my workstation. Most of this information was derived from
this <a href="http://www.microsoft.com/communities/newsgroups/list/en-us/default.aspx?dg=microsoft.public.windows.file_system&tid=4b1a14f7-6bd2-4c9f-ae64-df57c35712bf&cat=&lang=&cr=&sloc=&p=1">post</a>.</p>
<p>First step is to disable auto mounting of devices in Microsoft Vista by
running ‘cmd’ in an administrative user context and then execute
‘mountvol /N’ to enable readonly mounting of newly attached drives and volumes.</p>
<p><img alt="image" src="https://www.rsreese.com/assets/mountvol-729035.jpg"></p>
<p>Here is how to list the drives and volumes:</p>
<div class="highlight"><pre><span></span>DISKPART> list disk
Disk ### Status Size Free Dyn Gpt
-------- ---------- ------- ------- --- ---
Disk 0 Online 233 GB 0 B
Disk 1 Online 932 GB 0 B *
Disk 2 Online 932 GB 0 B *
Disk 3 No Media 0 B 0 B
Disk 4 Online 3911 MB 0 B
DISKPART> list vol
Volume ### Ltr Label Fs Type Size Status Info
---------- --- ----------- ----- ---------- ------- --------- --------
Volume 0 E DVD-ROM 0 B No Media
Volume 1 H BLACK_DAHLI UDF DVD-ROM 3214 MB Healthy
Volume 2 F U3 System CDFS CD-ROM 8 MB Healthy
Volume 3 C NTFS Partition 233 GB Healthy System
Volume 4 D data NTFS Partition 931 GB Healthy
Volume 5 Partition 931 GB Healthy
Volume 6 G Removable 0 B No Media
Volume 7 I FAT32 Removable 3911 MB Healthy
</pre></div>
<p><img alt="image2" src="https://www.rsreese.com/assets/readonly-removable-714947.jpg"></p>
<p>So I decided to try a spare drive in the system and I found that when attempting to mount a TrueCrypt volume I got an error telling me that auto-mount is not support and I would have to re-enable it.</p>
<p><img alt="image3" src="https://www.rsreese.com/assets/truecrypt-nomount-795930.jpg"></p>
<p>Continuing on my quest I was able to mount a spare hard drive volume read only, note you may also set the whole disk to read only.</p>
<div class="highlight"><pre><span></span>DISKPART> select volume 5
Volume 5 is the selected volume.
DISKPART> att vol set readonly
Volume attributes set successfully.
DISKPART> detail vol
Disk ### Status Size Free Dyn Gpt
-------- ---------- ------- ------- --- ---
* Disk 2 Online 932 GB 0 B *
Read-only : Yes
Hidden : No
No Default Drive Letter: Yes
Shadow Copy : No
Dismounted : Yes
BitLocker Encrypted : No
</pre></div>
<p>The next step will clear the read only status.</p>
<p><span class="caps">DISKPART</span>> att vol clear readonly<br>
Volume attributes cleared successfully.</p>
<p>Do not forget you may want to enable auto mounting again.</p>
<p>C:\Windows\system32>mountvol /N</p>
<p>A second and much easier alternative for <span class="caps">USB</span> devices is a small
application that changes a registry entry called <a href="http://www.irongeek.com/i.php?page=security/thumbscrew-software-usb-write-blocker">ThumbScrew</a>. It
alters a registry entry and though there is no guarantee that windows
still will not access the drive it is a quick fix for this scenario. My
plan is to use both methods. First disable the registry setting and then
using drive part set the read only flag.</p>
<p>If you have any ideas about mounting drives in a Windows environment
then please feel free to contact me and tell me about it.</p>Converting Microsoft OS to VMWare Guest2008-07-30T01:03:00-04:002008-07-30T01:03:00-04:00Stephen Reesetag:www.rsreese.com,2008-07-30:/converting-microsoft-os-to-vmware-guest/A friend had two notebooks running Microsoft XP Home and Professional editions in which the notebooks were no longer functional but the hard drives were in good shape so I recommend running them in a VM guest. I knew I could use VMWare converter tool that was freely available and …<p>A friend had two notebooks running Microsoft <span class="caps">XP</span> Home and Professional
editions in which the notebooks were no longer functional but the hard
drives were in good shape so I recommend running them in a <span class="caps">VM</span> guest. I
knew I could use VMWare converter tool that was freely available and it
supports converting from live hosts and images created from several
software programs. I was disappointed to find that VMWares converter
would not convert from Ghost enterprise (*.gho) images, but the latest
version of Symantec Norton Ghost 14.0 would so I created images of the drives.</p>
<p><img alt="Screen Shot" src="https://www.rsreese.com/assets/recoverypoint-733882.jpg"></p>
<p>After the images were created I next fired up VMWares converter and let perform its magic.</p>
<p><img alt="Screen Shot" src="https://www.rsreese.com/assets/vmconvert-734505.jpg"></p>
<p>This operation performed flawlessly. I ran both notebook
images with two hitches, I had to reactivate both <span class="caps">XP</span> installations
because running the guests inside VMWare workstation caused the
operating system to assume it was running a different hardware but this
wasn not a big deal. The second problem was trying to run the guest
operating systems in VMWares free server product. I received an error
message that the guest were created with more capabilities then what
VMWare server could handle so the friend decided to purchase the
workstation product in order to run the products.</p>Converting Microsoft Vista from one version to another2008-07-19T04:37:00-04:002008-07-19T04:37:00-04:00Stephen Reesetag:www.rsreese.com,2008-07-19:/converting-microsoft-vista-from-one-version-to-another/A desktop that I had which was used for work recently would not activate because it required connectivity to the companies KMS server which I would connect to via VPN to complete but since I no longer work there that is out of the question. Since the Vista OS was …<p>A desktop that I had which was used for work recently would not activate
because it required connectivity to the companies <span class="caps">KMS</span> server which I
would connect to via <span class="caps">VPN</span> to complete but since I no longer work there
that is out of the question. Since the Vista <span class="caps">OS</span> was an enterprise version
I had no way to purchase a license for it. I did however have a Vista
Business license that is legit so I wanted to migrate to it from the
version of Vista Enterprise.</p>
<p>First make sure that everything near and dear is backed up in case
something goes screwy.</p>
<p>Before inserting the Windows Vista <span class="caps">CD</span><br>
Go to, Start, Run: and type: regedit.exe<br>
Go to HKEY_LOCAL_MACHINE\<span class="caps">SOFTWARE</span>\Microsoft\Windows
<span class="caps">NT</span>\CurrentVersion<br>
Change the key : ProductName from “Windows Vista ™ Enterprise” to
“Windows Vista ™ Business”<br>
Change the key: EditionID from “Enterprise” to “Business”</p>
<p>Do not restart</p>
<p>Now insert Windows Vista <span class="caps">CD</span> and start upgrading (the option Upgrade will
not be graded out anymore)</p>
<p>A copy of program/drivers had to be reinstalled but much easier solution
for me then reinstalling everything which is usually a week long process
it seems like now.</p>Domain registrars spamming sub-domains?2008-07-03T02:31:00-04:002008-07-03T02:31:00-04:00Stephen Reesetag:www.rsreese.com,2008-07-03:/domain-registrars-spamming-sub-domains/In the process of setting up some virtual servers (slices) from www.slicehost.com I had to move the name servers around along with a migration to Google web apps. A user called complaining that they could not access the web-mail service. The user was trying to access www.mail …<p>In the process of setting up some virtual servers (slices) from
<a href="http://www.slicehost.com">www.slicehost.com</a> I had to move the name servers around along with a
migration to Google web apps. A user called complaining that they could
not access the web-mail service. The user was trying to access
www.mail.domain.com instead of mail.domain.com which a <span class="caps">DNS</span> record had
yet to be setup for and we weren’t planning on it. To our surprise there
was a page there though, a place holder with some nasty pop-ups. We
immediately added a record for this entry to kill it but it makes me
wonder how many other sub-domains have been compromised? The registrar
was <a href="http://www.godaddy.com">www.godaddy.com</a>, we will be migrating to a new one very soon.</p>Encrypting a secondary drive (PGP or TrueCrypt)2008-05-15T00:05:00-04:002008-05-15T00:05:00-04:00Stephen Reesetag:www.rsreese.com,2008-05-15:/encrypting-a-secondary-drive-pgp-or-truecrypt/In this post I am going to share my experiences with encrypting a secondary drive in a Windows Vista environment. The hardware is a Dell Optiplex core 2 duo. I will be encrypting a 1 terabyte Hitachi drive which I use primarily for storage. The first piece of software I …<p>In this post I am going to share my experiences with encrypting a
secondary drive in a Windows Vista environment.</p>
<p>The hardware is a Dell Optiplex core 2 duo. I will be encrypting a 1
terabyte Hitachi drive which I use primarily for storage.</p>
<p>The first piece of software I tried is <a href="http://www.pgp.com"><span class="caps">PGP</span> Desktop</a>. When setting up
the drives the first thing I noticed when partitioning them through
windows is I have a choice of boot record formats. As of this post <span class="caps">PGP</span>
Desktop did not even see a partition when a drive was initialized as
<a href="http://en.wikipedia.org/wiki/GUID_Partition_Table"><span class="caps">GPT</span></a> though it did not have a problem with the standard <a href="http://en.wikipedia.org/wiki/Mbr"><span class="caps">MBR</span></a> type.
I also attempted encrypting as a <span class="caps">MBR</span> type and then converting it to <span class="caps">GPT</span>.
<span class="caps">PGP</span> Desktop removed its encryption status when I did this therefore I
would not recommend trying that ;-). This concerned me since I am
planning on implementing a raid solution and do not want to be limited to
2 terabytes by the drive table type. Regardless I went with the <span class="caps">MBR</span>
style in order to allow <span class="caps">PGP</span> Desktop to play nicely. I imagine their
product will support the newer format in the future. Encrypting a
terabyte of data took all of the 12 hours for <span class="caps">AES</span>-256 which is what the
tell-tell meter said it would. Once encrypted it acted just like a
regular drive and upon restarting the Vista <span class="caps">OS</span> it prompted for a
pass-phrase. Pretty simple and clean.</p>
<p>On a side note when I broke <span class="caps">PGP</span> desktop encryption on the drive I had to
do the following to remove the bootguard since it resides on the boot drive:</p>
<p>Decrypting from a Command Line</p>
<ol>
<li>
<p>From the command line, type pgpwde —decrypt —disk 0 (or the disk
in question) —passphrase “enter passphrase here within double quotes”
and press the enter key. The disk will then decrypt. The <span class="caps">PGP</span> Whole Disk
status icon will be turning around in the system tray to show you
decryption is in progress:</p>
</li>
<li>
<p>Once decryption is complete, see if the disk is still instrumented
by bootguard by typing the —status command listed above. If the drive
is not encrypted, the hard drive should boot normally. If the drive is
still instrumented, but no highwater, proceed to the next steps.</p>
</li>
</ol>
<p><a href="http://www.truecrypt.org/">Truecrypt</a> was my next contestant. This appeals because of the great
support that many open source solutions provide from the community.
There are several algorithm options with TrueCrypt. I decide to go with
the <span class="caps">AES</span>-Serpent combination but benchmark was a little off though. When
creating the volume it also took around 10 hours for the terabyte volume
averaging about 25 <span class="caps">MB</span>/s which means the <span class="caps">AES</span> solo algorithm probably
would have taken half of the time.</p>
<p>I had some problems with the Truecrypt setup as well. The first round I
was warned about existing partitions so I deleted everything and let <span class="caps">TC</span>
encrypt the device (drive) instead of a partition which didn’t work so
well. I learned it is recommended to encrypt a partition instead of the
whole physical drive so I used the disk management snap-in via Vista’s
Administrative Tools to first create the partition using the <span class="caps">GPT</span> style
partition and let TrueCrypt format the drive using <span class="caps">NTFS</span>.</p>
<p>I have decided to stick with TrueCrypt over <span class="caps">PGP</span> Desktop because it’s
free and it let me use the <span class="caps">GPT</span> style partitioning scheme. There are
benefits to using <span class="caps">PGP</span>’s suite because it also includes email and instant
messaging encryption tools amongst others but there is a fee for using
the software beyond the demo period.</p>Force Outlook to open all email in plain text2008-02-12T03:51:00-05:002008-02-12T03:51:00-05:00Stephen Reesetag:www.rsreese.com,2008-02-12:/force-outlook-to-open-all-email-in-plain-text/For reference. Strip HTML email in Outlook into plain text Content: First, this is secure as many of the worms and bugs rely on HTML script code. One good example could be the needless advertisements or images sent inside spam (junk) emails. When you so much as view an email …<p>For reference.</p>
<p>Strip <span class="caps">HTML</span> email in Outlook into plain text Content: First, this is
secure as many of the worms and bugs rely on <span class="caps">HTML</span> script code. One good
example could be the needless advertisements or images sent inside spam
(junk) emails. When you so much as view an email inside your email
software, the senders webserver gets a timestamp of you having accessed
the image. This of course does not happen with plain text, because
there’s no image, so there is no inadvertent access?.</p>
<p>Second, it is also a bit faster to download and view email that doesn’t
have all the unnecessary frills of <span class="caps">HTML</span> email (tables, bold, italics etc).</p>
<p>Start | Run | regedit Find this key:
HKEY_CURRENT_USER\Software\Microsoft\Office\<br>
10.0\Outlook\Options\Mail On the Edit menu, point to New, and then
click DWord Value. With the new Dword value selected, type ReadAsPlain.
Double-click the new value to open it. In the Value Data box, type 1,
and then click <span class="caps">OK</span>. Click <span class="caps">OK</span>, and then quit Registry Editor. Just to be
sure, close Outlook and restart it. From now on, all your <span class="caps">HTML</span> email
messages will show up as simple text. After you turn on the Read as
Plain Text? feature, users notice the following changes:</p>
<p>The changes are applied to the preview pane and open messages. Pictures
become attachments to avoid loss. Digitally signed messages are not affected.</p>Disable fast user switching on Vista2008-02-12T03:49:00-05:002008-02-12T03:49:00-05:00Stephen Reesetag:www.rsreese.com,2008-02-12:/disable-fast-user-switching-on-vista/In Vista (unlike Windows XP), Fast User Switching works if you’re on a network domain. To turn off Fast User Switching, choose Start, type gpedit.msc in the Search box, and then press Enter. (If a security prompt appears, type an administrator password or confirm the action.) In the …<p>In Vista (unlike Windows <span class="caps">XP</span>), Fast User Switching works if you’re on a
network domain. To turn off Fast User Switching, choose Start, type
gpedit.msc in the Search box, and then press Enter. (If a security
prompt appears, type an administrator password or confirm the action.)
In the Group Policy Object Editor, choose</p>
<p>Local Computer Policy > Computer Configuration > Administrative Templates > System > Logon > enable Hide Entry Points for Fast User Switching > <span class="caps">OK</span>.</p>
<p>To find out who else is logged on to your computer:
1. Right-click an empty area of the taskbar and choose Task Manager. or Press Ctrl+Shift+Esc.
2. Click the Users tab to view users and their status</p>Kicking a user off a linux system2008-02-12T03:46:00-05:002008-02-12T03:46:00-05:00Stephen Reesetag:www.rsreese.com,2008-02-12:/kicking-a-user-off-a-linux-system/This might break something the user is doing. You have been warned. last -i1 baduser | awk '{print $3;exit}' | xargs -p --replace iptables -A INPUT -s {} -j drop if [ "`who | grep $1`" != "" ] ; then sid=`ps -jU $1 | awk '{print $3}' | tail -1`" kill -HUP $sid echo "$1 was logged in …<p>This might break something the user is doing. You have been warned.</p>
<div class="highlight"><pre><span></span>last -i1 baduser | awk '{print $3;exit}' | xargs -p --replace iptables -A INPUT -s {} -j drop if [ "`who | grep $1`" != "" ] ; then sid=`ps -jU $1 | awk '{print $3}' | tail -1`" kill -HUP $sid echo "$1 was logged in. Just booted $1 out." fi ps -u username | grep -v PID | awk '{print $1}' | xargs kill kill $(ps -u username | grep -v PID | awk '{print $1}')
</pre></div>Authenicating kerberos against active directory2008-02-12T03:37:00-05:002008-02-12T03:37:00-05:00Stephen Reesetag:www.rsreese.com,2008-02-12:/authenicating-kerberos-against-active-directory/Your /etc/pam.d/system-auth is created with the command “authconfig” on a RHEL5 machine though you may have to manually edit it with other distributions: #%PAM-1.0# This file is auto-generated. # User changes will be destroyed the next time authconfig is run. auth required /lib/security/$ISA/pam_env.so …<p>Your /etc/pam.d/system-auth is created with the command “authconfig” on
a <span class="caps">RHEL5</span> machine though you may have to manually edit it with other distributions:</p>
<div class="highlight"><pre><span></span>#%PAM-1.0# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth required /lib/security/$ISA/pam_env.so
auth sufficient /lib/security/$ISA/pam_unix.so like
auth nullokauth sufficient /lib/security/$ISA/pam_krb5.so use_first_pass
auth required /lib/security/$ISA/pam_deny.so
account required /lib/security/$ISA/pam_unix.so broken_shadow
account sufficient /lib/security/$ISA/pam_succeed_if.so uid < 100 quiet
account [default=bad success=ok user_unknown=ignore] /lib/security/$ISA/pam_krb5.so
account required /lib/security/$ISA/pam_permit.so
password requisite /lib/security/$ISA/pam_cracklib.so retry=3
password sufficient /lib/security/$ISA/pam_unix.so nullok use_authtok md5 shadow
password sufficient /lib/security/$ISA/pam_krb5.so use_authtok
password required /lib/security/$ISA/pam_deny.so
session required /lib/security/$ISA/pam_limits.so
session required /lib/security/$ISA/pam_unix.so
session optional /lib/security/$ISA/pam_krb5.so
</pre></div>
<p>Your /etc/krb5.conf should look something like this. Your system time
must be accurate or else it will not work correctly.</p>
<div class="highlight"><pre><span></span>[logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log[libdefaults] default_realm = AD.DOMAIN.EDUclockskew = 300 dns_lookup_realm = false dns_lookup_kdc = false ticket_lifetime = 24h forwardable = yes[realms]UFL.EDU = { kdc = DC01.AD.DOMAIN.EDU default_domain = DOMAIN.EDU }AD.DOMAIN.EDU = { kdc = ad.domain.edu admin_server = ad.domain.edu }[domain_realm] .domain.edu = DOMAIN.EDU domain.edu = DOMAIN.EDU[kdc] profile = /var/kerberos/krb5kdc/kdc.conf[appdefaults] pam = { debug = false ticket_lifetime = 36000 renew_lifetime = 36000 forwardable = true krb4_convert = false }
</pre></div>
<p>Next you need run kinit to make sure that you can contact the kerberos
server, if it returns nothing then you should be good.</p>
<div class="highlight"><pre><span></span>$ kinitPassword <span class="k">for</span> rsreese@AD.DOMAIN.EDU: blahblah
</pre></div>
<p>Next setup two cron entries to keep the time up to date and kinit
alive:<br>
$ sudo crontab -e</p>
<div class="highlight"><pre><span></span>0 23 * * 1,3,5 /usr/sbin/ntpdate time.nrc.ca0 */4 * * * kinit -R
</pre></div>
<p>The /etc/samba/smb.conf file needs to be setup.</p>
<div class="highlight"><pre><span></span># grep -Ev '#|;|^$' /etc/samba/smb.conf[global] workgroup = UFAD realm = AD.DOMAIN.EDU server string = SRVV-SERV hosts allow = 10.242. 10.228. load printers = no log file = /var/log/samba/%m.log max log size = 50 security = ads idmap uid = 10000 - 20000 idmap gid = 10000 - 20000winbind enum users=yeswinbind enum groups=yes template homedir = /home/%U template shell = /bin/bashclient use spnego = yes winbind use default domain = no encrypt passwords = yes smb passwd file = /etc/samba/smbpasswd socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192 local master = no dns proxy = no[homes] comment = %U Home Directory browseable = no path = %H valid users = %U writable = yes create mode = 0664 directory mode = 0775[printers] comment = All Printers path = /var/spool/samba browseable = no guest ok = no writable = no printable = yes
</pre></div>
<p>Now add the computer object to the domain via the Active directory
“Users and Computers”</p>
<p>You need to join the linux machine to the domain. First create an
account on the domain for the machine as mentioned in the beginning or
this will fail.<br>
# net ads join -U administrator</p>
<p>SElinux needs to be told to let Samba play nicely<br>
# setsebool -P samba_enable_home_dirs=1</p>
<p>\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~<span class="caps">NOT</span>
<span class="caps">NEEDED</span>\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~<br>
The /etc/ldap.conf looks like this:</p>
<p>host 10.241.28.100<br>
base dc=domain,dc=edu<br>
uri ldap://ad.domain.edu/<br>
binddn rsreese@domain.edu<br>
bindpw<br>
scope sub<br>
pam_filter objectclass=User<br>
pam_login_attribute sAMAccountName<br>
pam_lookup_policy yes<br>
nss_base_passwd dc=edu?sub<br>
nss_base_shadow dc=edu?sub<br>
nss_base_group dc=edu?sub<br>
nss_map_objectclass posixAccount user<br>
nss_map_objectclass shadowAccount user<br>
nss_map_attribute uid sAMAccountName<br>
nss_map_attribute homeDirectory unixHomeDirectory<br>
nss_map_attribute shadowLastChange pwdLastSet<br>
nss_map_objectclass posixGroup group<br>
nss_map_attribute uniqueMember member<br>
pam_login_attribute sAMAccountName<br>
pam_filter objectclass=User<br>
pam_password ad<br>
ssl no<br>
tls_cacertdir /etc/openldap/cacerts<br>
pam_password md5</p>
<p>\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~<span class="caps">NOT</span>
<span class="caps">NEEDED</span>\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~<br>
Next I edit the /etc/nsswitch.conf to add ldap support:</p>
<p>passwd: files ldap<br>
shadow: files<br>
group: files ldap<br>
hosts: files dns<br>
bootparams: nisplus [<span class="caps">NOTFOUND</span>=return] files<br>
ethers: files<br>
netmasks: files<br>
networks: files<br>
protocols: files<br>
rpc: files<br>
services: files<br>
netgroup: files<br>
publickey: nisplus<br>
automount: files<br>
aliases: files nisplus</p>Configuring sendmail to accept mail2008-02-12T03:34:00-05:002008-02-12T03:34:00-05:00Stephen Reesetag:www.rsreese.com,2008-02-12:/configuring-sendmail-to-accept-mail/if you get ( doing a netstat -an more ) tcp 0 0 127.0.0.1:25 0.0.0.0:\* LISTEN Then your sendmail server is configured to accept connections from localhost only. To change this behavior, you need to edit /etc/mail/sendmail.mc. Find the line that starts …<p>if you get ( doing a netstat -an more )</p>
<div class="highlight"><pre><span></span>tcp 0 0 127.0.0.1:25 0.0.0.0:\* LISTEN
</pre></div>
<p>Then your sendmail server is configured to accept connections from localhost only.</p>
<p>To change this behavior, you need to edit /etc/mail/sendmail.mc. Find the line that starts with DAEMON_OPTIONS ( suggest vi +/DAEMON_OPTIONS sendmail.mc ) and edit the field <code>Addr=</code> to change it to your <span class="caps">IP</span> Address.</p>
<p>Then go down approx. 7 lines and comment out the line that reads:</p>
<div class="highlight"><pre><span></span>FEATURE(\`accept\_unresolveable\_domains')dnl
</pre></div>
<p>Next, exit vi (or whatever editor you use) and do…</p>
<div class="highlight"><pre><span></span>m4 /etc/mail/sendmail.mc \> /etc/sendmail.cf
</pre></div>
<p>Restart sendmail, and you should be able to receive mail from other mhosts</p>Edit group policy on remote computer2008-02-12T03:32:00-05:002008-02-12T03:32:00-05:00Stephen Reesetag:www.rsreese.com,2008-02-12:/edit-group-policy-on-remote-computer/Want to open up the MMC of a local Group Policy on a remote machine? Simply go to Start Run and type: gpedit.msc /gpcomputer: Computername<p>Want to open up the <span class="caps">MMC</span> of a local Group Policy on a remote machine?</p>
<p>Simply go to Start Run and type:</p>
<div class="highlight"><pre><span></span>gpedit.msc /gpcomputer: Computername
</pre></div>Running processes in the background on Linux2008-02-12T03:30:00-05:002008-02-12T03:30:00-05:00Stephen Reesetag:www.rsreese.com,2008-02-12:/running-processes-in-the-background-on-linux/If you just want your program to simply run in the background, launch it with a “&” at the end of the command from the shell. However, if it expects to use stdout, stdin, or stderr, it will stop — so these must all be redirected to files or pipes. This will …<p>If you just want your program to simply run in the background, launch it
with a “&” at the end of the command from the shell. However, if it
expects to use stdout, stdin, or stderr, it will stop — so these must
all be redirected to files or pipes.</p>
<p>This will still leave it attached to the terminal and process group of
the shell, however. Thus you will not be able to log out of the command
prompt with the background jobs unless you detach them. To get around
this you can use the “nohup” and/or “setsid” commands when launching it.</p>
<p>If you want your program to daemonize itself (rather than relying on the
user to do it when invoking it), then you will have to read some unix
programming books about the steps involved. For example, Perl’s
Proc::Daemon does the following:</p>
<ol>
<li>
<p>Fork a child and exit the parent process.</p>
</li>
<li>
<p>Become a session leader (which detaches the program from the
controlling terminal).</p>
</li>
<li>
<p>Fork another child process and exit first child. This prevents the
potential of acquiring a controlling terminal.</p>
</li>
<li>
<p>Change the current working directory to “/”.</p>
</li>
<li>
<p>Clear the file creation mask.</p>
</li>
<li>
<p>Close all open file descriptors.</p>
</li>
</ol>Adding a character to a line using Perl2008-02-04T06:46:00-05:002008-02-04T06:46:00-05:00Stephen Reesetag:www.rsreese.com,2008-02-04:/adding-a-character-to-a-line-using-perl/perl -p -i -e 's/(.)$/$1$1/g' filename This changed my nonsense file: ghggk dethaks gjfkdld fyduftsdu flkgjd kflgjlk flkgjl f into a slightly different nonsense file: ghggkk dethakss gjfkdldd fyduftsduu flkgjd kflgjlk flkgjl ff<div class="highlight"><pre><span></span><span class="n">perl</span> <span class="o">-</span><span class="n">p</span> <span class="o">-</span><span class="n">i</span> <span class="o">-</span><span class="n">e</span> <span class="s">'s/(.)$/$1$1/g'</span> <span class="n">filename</span> <span class="n">This</span> <span class="n">changed</span> <span class="k">my</span> <span class="n">nonsense</span> <span class="n">file:</span> <span class="n">ghggk</span> <span class="n">dethaks</span> <span class="n">gjfkdld</span> <span class="n">fyduftsdu</span> <span class="n">flkgjd</span> <span class="n">kflgjlk</span> <span class="n">flkgjl</span> <span class="n">f</span> <span class="n">into</span> <span class="n">a</span> <span class="n">slightly</span> <span class="n">different</span> <span class="n">nonsense</span> <span class="n">file:</span> <span class="n">ghggkk</span> <span class="n">dethakss</span> <span class="n">gjfkdldd</span> <span class="n">fyduftsduu</span> <span class="n">flkgjd</span> <span class="n">kflgjlk</span> <span class="n">flkgjl</span> <span class="n">ff</span>
</pre></div>Getting Samba to play nicely with SELinux on RHEL2008-02-04T06:31:00-05:002008-02-04T06:31:00-05:00Stephen Reesetag:www.rsreese.com,2008-02-04:/getting-samba-to-play-nicely-with-selinux-on-rhel/This helpful bit was written by Don Meyer. I am a little too stubborn for a quick fix like this, so I went the route of adding the specific rules needed to allow SMB/Winbindd to run without throwing AVC errors. I am doing this on RHEL4 boxes, which install …<p>This helpful bit was written by Don Meyer.</p>
<p>I am a little too stubborn for a quick fix like this, so I went the<br>
route of adding the specific rules needed to allow <span class="caps">SMB</span>/Winbindd to<br>
run without throwing <span class="caps">AVC</span> errors. I am doing this on <span class="caps">RHEL4</span> boxes,<br>
which install with SElinux enforcing targeted by default — this<br>
allows me to leave SElinux active for its additional protections.</p>
<p>Doing it this way requires a little extra work, though…</p>
<p>First, you need to install the selinux-policy-targeted-sources<br>
package, if not already installed.</p>
<p>When I build the RPMs from the source tarball, the first upgrade from<br>
the default <span class="caps">RHEL4</span> packages changes the tdb directory from<br>
/var/cache/samba/ to /var/lib/samba/. This is accomplished by<br>
creating /var/lib/samba/ — Naturally, this royally mucks up the<br>
SElinux labelings/permissions. So, immediately after the first<br>
upgrade from <span class="caps">RHEL4</span> samba packages, (before starting either smb or<br>
winbind) I need to do the following:</p>
<div class="highlight"><pre><span></span>#chcon -Rt samba_var_t /var/lib/samba
#mkdir /var/lib/samba/winbindd_privileged/
#chcon -t winbind_var_run_t /var/lib/samba/winbindd_privileged/
</pre></div>
<p>Then, I drop the following file into the directory<br> /etc/selinux/targeted/src/policy/domains/misc/:</p>
<p>winbind_add.te:</p>
<div class="highlight"><pre><span></span><span class="nt">allow</span> <span class="nt">winbind_t</span> <span class="nt">etc_runtime_t</span><span class="p">:</span><span class="nd">file</span> <span class="nt">read</span><span class="o">;</span>
<span class="nt">allow</span> <span class="nt">winbind_t</span> <span class="nt">proc_t</span><span class="p">:</span><span class="nd">file</span> <span class="nt">read</span><span class="o">;</span>
<span class="nt">allow</span> <span class="nt">winbind_t</span> <span class="nt">etc_t</span><span class="p">:</span><span class="nd">file</span> <span class="nt">write</span><span class="o">;</span>
<span class="nt">allow</span> <span class="nt">winbind_t</span> <span class="nt">samba_etc_t</span><span class="p">:</span><span class="nd">file</span> <span class="nt">write</span><span class="o">;</span>
<span class="nt">allow</span> <span class="nt">winbind_t</span> <span class="nt">initrc_t</span><span class="p">:</span><span class="nd">process</span> <span class="p">{</span> <span class="err">signal</span> <span class="err">signull</span> <span class="p">}</span><span class="o">;</span>
<span class="nt">allow</span> <span class="nt">winbind_t</span> <span class="nt">initrc_var_run_t</span><span class="p">:</span><span class="nd">file</span> <span class="p">{</span> <span class="err">lock</span> <span class="err">read</span> <span class="p">}</span><span class="o">;</span>
<span class="nt">allow</span> <span class="nt">winbind_t</span> <span class="nt">var_lib_t</span><span class="p">:</span><span class="nd">dir</span> <span class="p">{</span> <span class="err">search</span> <span class="err">getattr</span> <span class="p">}</span><span class="o">;</span>
<span class="nt">allow</span> <span class="nt">winbind_t</span> <span class="nt">var_lib_t</span><span class="p">:</span><span class="nd">dir</span> <span class="nt">search</span><span class="o">;</span>
<span class="nt">allow</span> <span class="nt">winbind_t</span> <span class="nt">samba_log_t</span><span class="p">:</span><span class="nd">dir</span> <span class="p">{</span> <span class="err">create</span> <span class="err">setattr</span> <span class="p">}</span><span class="o">;</span>
<span class="nt">allow</span> <span class="nt">winbind_t</span> <span class="nt">unconfined_t</span><span class="p">:</span><span class="nd">fifo_file</span> <span class="nt">read</span><span class="o">;</span>
<span class="nt">allow</span> <span class="nt">winbind_t</span> <span class="nt">var_lib_t</span><span class="p">:</span><span class="nd">dir</span> <span class="nt">search</span><span class="o">;</span>
</pre></div>
<p>This file is what I currently need to add to the default SElinux<br>
configuration to get Samba 3.0.23pre1 to work. What is needed seems<br>
to change with each new version of Samba… (The default SElinux<br>
ruleset for 3.0.10-1.3E.6 can be found in<br> “/etc/selinux/targeted/src/policy/domains/program/winbind.te”.)</p>
<p>Finally, after this “extra” policy file is in place, you should chdir<br>
to “/etc/selinux/targeted/src/policy/”, and run the following command:</p>
<p>#make load</p>
<p>After this, you should be able to start/restart the smb <span class="amp">&</span> winbind<br>
services without complaints.</p>
<p>Now, some might ask “How do you derive these additional rules?”</p>
<p>On a clean install, I install the packages, make the necessary mods,<br>
and then set SElinux to non-enforcing:</p>
<p>#setenforce 0</p>
<p>I then start “tail -f /var/log/messages > /tmp/samba_avc.log” in a<br>
separate console.</p>
<p>Next, I start the smb <span class="amp">&</span> winbind services and get the running<br>
properly. Running in non-enforcing mode allows all the error<br>
messages to be generated in the logs, but the operations are allowed<br>
to complete successfully. Once the services are running, I do a<br>
couple user queries to prime the winbind system and have it sync with<br>
the <span class="caps">AD</span>, etc. I then terminate the tail in the other console, and run<br>
the following command:</p>
<p>#audit2allow -i /tmp/samba_avc.log</p>
<p>This outputs (to stdout) the additional rules necessary to allow all<br>
of the operations that generated <span class="caps">AVC</span> error messages in the log<br>
excerpt. This should be what is necessary to get everything running<br>
— I copy these rules into the file I call winbind_add.te in<br>
“/etc/selinux/targeted/src/domains/misc/”, and run the “make load”<br>
command to force the system to reload the SElinux rules.</p>
<p>Finally, I can shut down the smb <span class="amp">&</span> winbind services, run “setenforce<br>
1” to re-enable SElinux enforcing mode, and then restart smb &<br>
winbind. If all goes well, this should not generate any <span class="caps">AVC</span> errors…</p>Remove index.php from wiki URL2008-02-04T06:22:00-05:002008-02-04T06:22:00-05:00Stephen Reesetag:www.rsreese.com,2008-02-04:/remove-index-php-from-wiki-url/In httpd.conf: Alias /wiki/index.php /home/rsreese/richardsreese/htdocs/w/index.php Alias /wiki /home/rsreese/richardsreese/htdocs/w/index.php In Localsetting.php: $wgScriptPath = "/w"; $wgScript = "$wgScriptPath/index.php"; $wgRedirectScript = "$wgScriptPath/redirect.php"; # For more information on customizing the URLs please see: # http://meta.wikimedia.org/wiki …<p>In httpd.conf:</p>
<div class="highlight"><pre><span></span>Alias /wiki/index.php /home/rsreese/richardsreese/htdocs/w/index.php
Alias /wiki /home/rsreese/richardsreese/htdocs/w/index.php
</pre></div>
<p>In Localsetting.php:</p>
<div class="highlight"><pre><span></span>$wgScriptPath = "/w";
$wgScript = "$wgScriptPath/index.php";
$wgRedirectScript = "$wgScriptPath/redirect.php";
# For more information on customizing the URLs please see:
# http://meta.wikimedia.org/wiki/Eliminating\_index.php\_from\_the\_url
# If using PHP as a CGI module, the ?title= style usually must be used.
#$wgArticlePath = "$wgScript/$1";
$wgArticlePath = "/wiki/$1";
</pre></div>Courier Vacation Notice2008-02-04T06:09:00-05:002008-02-04T06:09:00-05:00Stephen Reesetag:www.rsreese.com,2008-02-04:/courier-vacation-notice/cc "| /usr/lib/courier/bin/mailbot -t autoresponse -s 'AutoAwayMessage' -A 'From: test@somedomain.com' /usr/sbin/sendmail -f ''"cc "!user@somedomain.edu"cc "./Maildir" EXITCODE = 0 exit<div class="highlight"><pre><span></span>cc "| /usr/lib/courier/bin/mailbot -t autoresponse -s 'AutoAwayMessage' -A 'From: test@somedomain.com' /usr/sbin/sendmail -f ''"cc "!user@somedomain.edu"cc "./Maildir"
EXITCODE = 0
exit
</pre></div>Compare Directory Contents on Linux computer2008-02-01T02:02:00-05:002008-02-01T02:02:00-05:00Stephen Reesetag:www.rsreese.com,2008-02-01:/compare-directory-contents-on-linux-computer/#/bin/bash DIR_1=$1 DIR_2=$2 #check dir diffs ls -1 $DIR_1 >/tmp/diff.1 ls -1 $DIR_2 >/tmp/diff.2 echo "Check Dir differences:" diff /tmp/diff.1 /tmp/diff.2 && echo "Dir's have the same files" #check files differences echo "check files differences:" for file in `cat …<div class="highlight"><pre><span></span><span class="c1">#/bin/bash</span>
<span class="nv">DIR_1</span><span class="o">=</span><span class="nv">$1</span>
<span class="nv">DIR_2</span><span class="o">=</span><span class="nv">$2</span>
<span class="c1">#check dir diffs</span>
ls -1 <span class="nv">$DIR_1</span> >/tmp/diff.1
ls -1 <span class="nv">$DIR_2</span> >/tmp/diff.2
<span class="nb">echo</span> <span class="s2">"Check Dir differences:"</span>
diff /tmp/diff.1 /tmp/diff.2 <span class="o">&&</span> <span class="nb">echo</span> <span class="s2">"Dir's have the same files"</span>
<span class="c1">#check files differences</span>
<span class="nb">echo</span> <span class="s2">"check files differences:"</span>
<span class="k">for</span> file in <span class="sb">`</span>cat /tmp/diff.1 /tmp/diff.2<span class="p">|</span>uniq<span class="sb">`</span><span class="p">;</span> <span class="k">do</span>
diff <span class="nv">$DIR_1</span>/<span class="nv">$file</span> <span class="nv">$DIR_2</span>/<span class="nv">$file</span> <span class="m">2</span>>/dev/null
<span class="k">done</span>
rm /tmp/diff.1 /tmp/diff.2
</pre></div>NFS howto with static ports2008-01-31T06:43:00-05:002008-01-31T06:43:00-05:00Stephen Reesetag:www.rsreese.com,2008-01-31:/nfs-howto-with-static-ports/First I am going to edit the /etc/sysconfig/nfs to specify the ports I want to run on. STATD_PORT=4000 STATD_OUTGOING_PORT=4004 LOCKD_TCPPORT=4001 LOCKD_UDPPORT=4001 MOUNTD_PORT=4002 Next I want to edit the /etc/hosts.allow to only allow specific hosts to access the resource. nfs:192.168 …<p>First I am going to edit the /etc/sysconfig/nfs to specify the ports I
want to run on.</p>
<div class="highlight"><pre><span></span>STATD_PORT=4000
STATD_OUTGOING_PORT=4004
LOCKD_TCPPORT=4001
LOCKD_UDPPORT=4001
MOUNTD_PORT=4002
</pre></div>
<p>Next I want to edit the /etc/hosts.allow to only allow specific hosts to
access the resource.</p>
<div class="highlight"><pre><span></span><span class="n">nfs</span><span class="o">:</span><span class="mf">192.168</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span>
</pre></div>
<p>Finally lets allow some stuff to come in through our <span class="caps">IP</span> tables rules at /etc/sysconfig/iptables</p>
<div class="highlight"><pre><span></span>*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
#-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 137:139 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 111 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 2049 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 4000:4004 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 55443 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-port-unreachable
COMMIT
</pre></div>Multi-Touch Techonolgy - The new interface of computing2007-05-30T17:08:00-04:002007-05-30T17:08:00-04:00Stephen Reesetag:www.rsreese.com,2007-05-30:/multi-touch-techonolgy-the-new-interface-of-computing/This video shows some of the capabilities of this system.<p>This video shows some of the capabilities of this system.</p>
<p><object width="416" height="342" classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=6,0,40,0"><param name="pluginspage" value="http://www.macromedia.com/shockwave/download/index.cgi?P1_Prod_Version="></param><param name="src" value="http://services.brightcove.com/services/viewer/federated_f8/271552687"></param><param name="swliveconnect" value="true"></param><param name="seamlesstabbing" value="false"></param><param name="base" value="http://admin.brightcove.com"></param><param name="flashvars" value="videoId=933742930&playerId=271552687&viewerSecureGatewayURL=https://services.brightcove.com/services/amfgateway&servicesURL=http://services.brightcove.com/services&cdnURL=http://admin.brightcove.com&domain=embed&autoStart=false&"></param><embed width="416" height="342" type="application/x-shockwave-flash" src="http://services.brightcove.com/services/viewer/federated_f8/271552687" pluginspage="http://www.macromedia.com/shockwave/download/index.cgi?P1_Prod_Version=" swliveconnect="true" seamlesstabbing="false" base="http://admin.brightcove.com" flashvars="videoId=933742930&playerId=271552687&viewerSecureGatewayURL=https://services.brightcove.com/services/amfgateway&servicesURL=http://services.brightcove.com/services&cdnURL=http://admin.brightcove.com&domain=embed&autoStart=false&"></embed></object></p>SQL injection attack on a PostgreSQL database (t_jiaozhu)2007-03-28T15:22:00-04:002007-03-28T15:22:00-04:00Stephen Reesetag:www.rsreese.com,2007-03-28:/sql-injection-attack-on-a-postgresql-database-t_jiaozhu/A web server running Apache 2 and PostgreSQL was successfully compromised using a SQL injection vulnerability. I first noticed there was a new table in one of our PostgreSQL databases named ‘t_jiaozhu’. public t\_jiaozhu table postgres The table wasn not something that myself or our developer had created so …<p>A web server running Apache 2 and <a href="http://www.postgresql.org/">PostgreSQL</a> was successfully
compromised using a <span class="caps">SQL</span> injection vulnerability. I first noticed there
was a new table in one of our PostgreSQL databases named ‘t_jiaozhu’.</p>
<div class="highlight"><pre><span></span>public t\_jiaozhu table postgres
</pre></div>
<p>The table wasn not something that myself or our developer had created so I
immediately went into <span class="caps">WTF</span> mode. First I googled for the term ‘t_jiazhu’
and found that there was only one English result that mentioned <span class="caps">SQL</span>
injection attacks with the previously mentioned table name. At this
point we searched the PostgreSQL log files but did not turn up much but
with the advice of our local security engineer. We checked out the
Apache web server log files and found the attack.</p>
<div class="highlight"><pre><span></span><span class="err">#</span> <span class="nt">grep</span> <span class="nt">t</span><span class="err">\</span><span class="nt">_jiaozhu</span> <span class="err">\</span><span class="o">*</span><span class="nt">fred-access</span><span class="err">\</span><span class="nt">_log</span><span class="p">:</span><span class="nd">219</span><span class="p">.</span><span class="nc">153</span><span class="p">.</span><span class="nc">131</span><span class="p">.</span><span class="nc">99</span> <span class="nt">-</span> <span class="nt">-</span> <span class="cp">[</span><span class="mi">25</span><span class="p">/</span><span class="nx">Mar</span><span class="p">/</span><span class="nx">2007</span><span class="p">:</span><span class="mi">11</span><span class="p">:</span><span class="mi">59</span><span class="p">:</span><span class="mi">32</span> <span class="o">-</span><span class="mi">0400</span><span class="cp">]</span> <span class="s2">"HEAD /showemploymentopportunity.php?id=38;create%20table%20t\_jiaozhu(jiaozhu%20varchar(200)) HTTP/1.1"</span> <span class="nt">200</span> <span class="nt">-</span> <span class="s2">"-"</span> <span class="s2">"Mozilla/3.0 (compatible; Indy Library)"</span>
</pre></div>
<p>The engineer also came up with a possibility that the <span class="caps">IP</span> in which the
attack came from may have been a bot using an <a href="http://en.wikipedia.org/wiki/Intrusion-detection_system"><span class="caps">IDS</span></a>.</p>
<blockquote>
<p><span class="dquo">“</span>After the table was created, there were several hits from that <span class="caps">IP</span>
that had the following user agent “Mozilla/3.0 (compatible; Indy
Library)”. A little digging shows that it might be a Chinese spambot.”</p>
</blockquote>
<p>Our developer quickly discovered that we were not checking variables that
were being passed. A quick addition of code fixed the problem.</p>
<div class="highlight"><pre><span></span>if (!is\_numeric($id))
$id = 0;
</pre></div>Running UAC and some other tricks to keep your computer running smoothly2007-03-07T04:47:00-05:002007-03-07T04:47:00-05:00Stephen Reesetag:www.rsreese.com,2007-03-07:/running-uac-and-some-other-tricks-to-keep-your-computer-running-smoothly/Most users I know run Microsoft products. A few of you may benefit from some basic tips to keep your computer out of BestBuy or your local computer vendor for repairs. The first and probably most important is also the most difficult to get people to abide by. Use UAC …<p>Most users I know run <a href="http://www.microsoft.com/">Microsoft</a> products. A few of you may
benefit from some basic tips to keep your computer out of <a href="http://www.bestbuy.com/">BestBuy</a> or
your local computer vendor for repairs. The first and probably most
important is also the most difficult to get people to abide by. Use
<a href="http://technet.microsoft.com/en-us/windowsvista/aa906021.aspx"><span class="caps">UAC</span></a> (user access controls). By default Windows <span class="caps">XP</span> uses the
administrator account which is convenient when an operating system is
first loaded but most users load all of their programs on a <span class="caps">PC</span> in just a
short time. After you get everything installed run as a ‘user’ account
and not an administrative context. This will prevent most spy ware and
viruses from trashing your system. Even if you accidentally download
some malware it will most likely at the worst trash the user profile but
not the system which is a pretty easy fix.</p>
<p>Vista by default has <span class="caps">UAC</span> turned on. This is annoying at first but is a
positive action by Microsoft in order to cut down on end-users trashing
their systems. <span class="caps">UAC</span> may be disabled but I wouldn’t recommend it. A
majority of computers that become compromised with spy ware is because
malware or viruses entered through a profile that had administrative
privileges and then self installed.</p>
<p><a href="http://en.wikipedia.org/wiki/Antivirus_software">Antivirus</a> must be installed. Most computers I come across don’t have
it installed or it’s so out of date it might as well not be installed.
It’s a small fee to pay or even <a href="http://free.grisoft.com/doc/1">free</a> to avoid the headache of
infecting your computer or worse other computers.</p>
<p>Scripting attacks may be prevented by staying out of crappy sites. One
problem is some popular sites still seem to host ads from vendors that
are known to install malware. Using a <a href="http://www.spywarewarrior.com/uiuc/resource.htm#IESPYAD">registry</a> based block lists is
a quick and free way to avoid these pitfalls.</p>
<p><a href="http://en.wikipedia.org/wiki/Peer-to-peer">Peer2Peer</a> software is another way to trash a system. Installing
poorly written software for the purpose of downloading music and whatnot
is a pretty sure fire way to hose a system. While in college most of the
computers I have seen that run poorly are because a Napster type of
software was installed and some of the files downloaded from the network
were virus ridden. The peer sharing software themselves sometimes have
ad-ware built in for the purpose of bombarding your computer with trash.
So the alternative sucks but pay for it using iTunes or something along
those lines.</p>
<p>With regards to email, if it looks too good to be true then it probably is. Do not click on links or download images from it, just delete and/or report it as <a href="http://en.wikipedia.org/wiki/Spam_%28electronic%29">spam</a>.</p>Running Terminal Server on Windows 2003 Server2007-02-08T05:33:00-05:002007-02-08T05:33:00-05:00Stephen Reesetag:www.rsreese.com,2007-02-08:/running-terminal-server-on-windows-2003-server/Vista has been a decent Operating System so far but there are still a large number of software vendors who were not prepared for the OS. A number of statistical software packages are at this point not supported so I decided to implement a Terminal Server for users to access …<p>Vista has been a decent Operating System so far but there are still a large number of software vendors who were not prepared for the <span class="caps">OS</span>. A number of statistical software packages are at this point not supported so I decided to implement a Terminal Server for users to access. The terminal server is not being deployed only as a quick fix to manufacturers short comings in software development. I have made the server available on a <span class="caps">VPN</span> for users to work from home where they do not have access to applications that are usually required to run on a <span class="caps">LAN</span>. Maintenance, licensing, and performance are some of the other benefits.</p>
<p>The first trick to setting up the terminal server was licensing. Since we are not running a cluster of terminal servers the license model was simple. I was able to set the terminal server to be a license server for its self which saved me from having to setup another machine to be a license server. Next was a journey over to <span class="caps">CDW</span> in order to purchase some terminal server licenses. When setting up the server there are two license modes, per device and per user. I went with per user because I wanted several hundred users to be able to login without having several hundred licenses.</p>
<p>Next was to setup security on the server so that only the groups I wanted would be able to login. Group policies were also implemented so that folder redirection and additional security features could be employed. The users must login through a vpn from remote locations though with most of our users have fast Internet connections so the vpn didn not really cause additional latency. Documentation was the final product to be constructed. As with any documentation I have gotten feedback to help write enough information so that all of the users are able to be instructed how to connect to our server and run applications remotely.</p>Using Common Sense to Secure your Information2006-12-20T05:53:00-05:002006-12-20T05:53:00-05:00Stephen Reesetag:www.rsreese.com,2006-12-20:/using-common-sense-to-secure-your-information/Every day technology creates efficiency for millions of people. With all of the benefits that technology provides there are also many pitfalls that come with convenience. Online vendors make it easy for people to purchase goods at reasonable prices when compared to brick and mortar stores. There are many good …<p>Every day technology creates efficiency for millions of people. With all of the benefits that technology provides there are also many pitfalls that come with convenience. Online vendors make it easy for people to purchase goods at reasonable prices when compared to brick and mortar stores. There are many good companies to do business with but there are also a lot of shady vendors. There are some pretty easy ways to spot the malicious vendors.</p>
<p>A site that looks poorly designed can be a sign of a site that was put together with haste just to be taken down shortly after a few people are ripped off. Searching for reviews of the company that people have expressed there opinions similar to what <a href="http://www.ebay.com">eBay</a> has in the form of feedback may help you decide. Also companies that are serious about business will no doubt have thoroughly thought about security and usually their reputation.</p>
<p>Do not use sites from <a href="http://en.wikipedia.org/wiki/Spam_(electronic)"><span class="caps">SPAM</span></a> or other illegitimate sources. <a href="http://en.wikipedia.org/wiki/Phishing">Phishing</a> sites are a sure fire way to have your identity stolen and you do not want that to happen. Make sure the site uses an <a href="http://en.wikipedia.org/wiki/Secure_Sockets_Layer"><span class="caps">SSL</span></a> certificate in order to encrypt your information, this is a must have. Do not use the same password for your various logins at different sites. Use at least 8 characters if not more and make sure to include some random characters which make <a href="http://en.wikipedia.org/wiki/Password_cracking">cracking</a> a password much more difficult. Know that there are sites that you may login to that may not use <span class="caps">SSL</span> certificates so your password may be picked up using a traffic sniffer. Also wireless networks are an easy way to lose information. Be weary of people listening on the wire with <a href="http://en.wikipedia.org/wiki/Packet_sniffer">traffic sniffers</a>. Do not send important information via email and instant messengers since they are almost always sent in clear text. Review you credit at least once a year, you may not even know that you a victim of online or identity theft until it has already happened.</p>
<p>There are a number of resources online to help you from online fraud. A simple Google search can help you find these resources.</p>Microsoft Vista and Office 2007 Initial Review2006-12-12T03:58:00-05:002006-12-12T03:58:00-05:00Stephen Reesetag:www.rsreese.com,2006-12-12:/microsoft-vista-and-office-2007-initial-review/I recently got my hands on a copy of Microsoft’s latest offering in the form of desktop software, Vista and Office 2007. I have also acquired some new 64 bit Core 2 Duo Dell computers in order to test the new software for deployment though I have also been …<p>I recently got my hands on a copy of <a href="http://www.microsoft.com/">Microsoft’s</a> latest offering in
the form of desktop software, <a href="http://www.microsoft.com/windowsvista/">Vista</a> and <a href="http://office.microsoft.com/en-us/products/default.aspx?ofcresset=1">Office 2007</a>. I have also
acquired some new 64 bit <a href="http://www.intel.com/products/processor/core2duo/index.htm">Core 2 Duo</a> Dell computers in order to test
the new software for deployment though I have also been testing the new
offerings on older hardware in order to determine which machines will
need to be depreciated in the next year or two.</p>
<p>First I went ahead installed Office 2007 on my Windows <span class="caps">XP</span> desktop. As
with most Office installs I was able to customize an install file so
that I can skip on the license agreements, serial number and all of the
other annoying stuff. I’m pretty impressed overall with the office
install. The look of Office has been improved to use a ‘ribbon’
interface which is to improve productivity. Many users have already had
issues using the “Office Button” which incorporates many of the
functions that “file” button previously did. This is a common hang up
with major releases from a software vendor; end-users will have to take
time to become acclimated with the new functions. A trick feature I just
picked up on recently was just hitting the “alt” key will highlight the
shortcut’s to all of the current functions on the “ribbon” toolbar.</p>
<p>Vista was next on the list for testing. From the start I figured the
install would be large since we had to rip the <a href="http://en.wikipedia.org/wiki/ISO_image"><span class="caps">ISO</span> image</a> to a
<a href="http://en.wikipedia.org/wiki/DVD"><span class="caps">DVD</span></a>. We started off with a 1.8 <a href="http://www.amd.com/us-en/"><span class="caps">AMD</span></a> with 512 <span class="caps">MB</span> of system memory.
I knew running a video card with 64 <span class="caps">MB</span> of memory would limit the
operating systems’ capability graphics wise but I needed a real world
baseline in which Vista could run without aggrevating end-users with
slowly responding applications. The install was very simple although I
did provide a answer file so I wouldn’t have to bother with serial
numbers and whatnot. Once Vista was up and running I was happy with the
performance overall for the base install. Next I added a beta version of
<a href="http://www.mcafee.com/us/">McAfee</a> antivirus for Vista, Office 2007, and some statistical
software such as <span class="caps">SAS</span>, Gams, Guass, and Limdep. The machine did slowdown
somewhat mainly due to background services and the lack of memory didn’t
help things much but this did give me a baseline for which machines
would be able to handle Vista performance wise.</p>
<p>Next was the 64 bit Vista install on 2.4 GHz Core 2 Duo chips, 1
gigabyte of memory, and 512 <span class="caps">MB</span> of video memory. These machines are
amazing, Vista of course allows for the full blown user interface
including Aero which provides for some pretty cool eye candy. I was able
to load this machine down and it wasn’t phased at all. For a $1000
dollars (not including monitor) these machines are going to be the way
to go for user’s that want the full Vista experience.</p>
<p>The final test to make Vista useable was to add it to the domain. I was
able to add the machines to the domain without a hiccup. Setting up
Outlook with the <a href="http://www.microsoft.com/exchange/default.mspx">Exchange</a> server was even easier since it picked up
the domain credentials from the currently logged in user. That is where
the fun ended. Vista employs <a href="http://technet.microsoft.com/en-us/windowsvista/aa906021.aspx">User Access Controls</a> (<span class="caps">UAC</span>) so the
domain policy’s made software installation rather annoying at least. The
lab computers were even worse because we log users in as guests so
profiles are not stored eating up drive space. Vista applies the group
policies to all accounts, even accounts that are not on the domain so
the only fix was to move a computer out of the <a href="http://en.wikipedia.org/wiki/Organizational_Unit">organizational unit</a>
(<span class="caps">OU</span>) before installing software so the restrictions aren’t there and
then moving it back in when done.</p>
<p>In summary I am impressed with Vista (with the right hardware) but have
a lot of tooling to do in order to find all of the benefits. I figure a
desktop computer with a 2 GHz processor, 512 Mb system memory, and 128
Mb video memory should be the baseline for us.</p>Copyrighted Music and Movies2006-10-18T05:06:00-04:002006-10-18T05:06:00-04:00Stephen Reesetag:www.rsreese.com,2006-10-18:/copyrighted-music-and-movies/Ever since the Napster rise and fall there has been an on going debate in regards to copyrighted material being shared across networks with peer to peer (P2P) applications and popular social networking websites. I know from my school and work that technology exists that may analyze network traffic and …<p>Ever since the Napster rise and fall there has been an on going debate
in regards to copyrighted material being shared across networks with
peer to peer (<span class="caps">P2P</span>) applications and popular social networking websites.
I know from my school and work that technology exists that may analyze
network traffic and determine what content travels through a connection.
The content may be stopped if deemed a violation of copyrighted materials. For example, if a student is transferring a song or video from home to their email account so they may upload it to their Ipod. Corporate employees may be exempt from many of the free speech debates that arise. Universitys on the other hand, at least public universitys have large student bodies to please, and furthermore these students have rights. The technology can be very expensive if a <a href="http://www.eff.org/share/?f=audible_magic.html">third
party</a> is used to thwart sharing of copyrighted materials.</p>
<p>Another hot topic are the social networking sites such as
<a href="http://myspace.com">myspace.com</a> and <a href="http://youtube.com">youtube.com</a> which contain quite a bit of
copyrighted material. The content is placed on the sites and shared by
the person users but ultimately the site is distributing the music. The
music and videos help a lot of newer bands that are just starting gain
popularity without spending tons of money on advertising. The same
technology that may be used on college and corporate network may also be
used on the networks that have web servers that distribute non-copyrighted material in order to find items that should not be shared.</p>
<p>A final interesting note for those who do not pay attention to the news
(of any sort), <a href="http://google.com">google.com</a>purchased youtube.com. This move for Google
is a huge step since they spent 1.85 billion dollars on youTube which is
already having issues due to the amount of copyrighted material that
the artists are complaining about.</p>What is Web 2.02006-04-18T19:24:00-04:002006-04-18T19:24:00-04:00Stephen Reesetag:www.rsreese.com,2006-04-18:/what-is-web-2-0/An article describing the slow migration to what some call Web 2.0<p>An <a href="http://www.oreillynet.com/pub/a/oreilly/tim/news/2005/09/30/what-is-web-20.html?page=1">article</a> describing the slow migration to what some call Web 2.0</p>Botnets that make money but at whos expense2006-03-22T19:08:00-05:002006-03-22T19:08:00-05:00Stephen Reesetag:www.rsreese.com,2006-03-22:/botnets-that-make-money-but-at-whos-expense/Witlog claims he do not use his botnet for illegal purposes, only “for fun.” I found that claim pretty hard to believe given a) the income he could make installing ad-serving software on each computer under his control, combined with b) the risk he is taking of getting caught breaking …<p>Witlog claims he do not use his botnet for illegal purposes, only “for
fun.” I found that claim pretty hard to believe given a) the income he
could make installing ad-serving software on each computer under his
control, combined with b) the risk he is taking of getting caught
breaking into so many computers. The kid I wrote about in the <a href="http://www.washingtonpost.com/wp-dyn/content/article/2006/02/14/AR2006021401342.html">Post
magazine story on the connection between botnets and spyware</a> was
making $6,000 to $10,000 per month installing adware on a botnet half
the size of the one Witlog claims to have.</p>New phishing techniques to fool online users2006-02-14T07:04:00-05:002006-02-14T07:04:00-05:00Stephen Reesetag:www.rsreese.com,2006-02-14:/new-phishing-techniques-to-fool-online-users/People are becoming aware of the insecurities posed by online shopping, browsing, and even messaging. The days of email that are obviously spam due to misspelled words and links that contain ip addresses instead of dns names are moving to a new level. The following post describes the process in …<p>People are becoming aware of the insecurities posed by online shopping, browsing, and even messaging. The days of email that are obviously spam due to misspelled words and links that contain ip addresses instead of dns names are moving to a new level. The following <a href="http://blog.washingtonpost.com/securityfix/2006/02/the_new_face_of_phishing_1.html">post</a> describes the process in which a <span class="caps">SSL</span> certificate was used to trick users into entering confidential information, a tatic previously not used before. </p>