<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Posts on SecopsMonkey</title>
    <link>https://secopsmonkey.com/post/</link>
    <description>Recent content in Posts on SecopsMonkey</description>
    <generator>Hugo -- gohugo.io</generator>
    <language>en-us</language>
    <lastBuildDate>Mon, 06 Apr 2026 11:41:19 -0400</lastBuildDate><atom:link href="https://secopsmonkey.com/post/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Tool Library Announcement</title>
      <link>https://secopsmonkey.com/post/2026-04-06-tool-library-announcement/</link>
      <pubDate>Mon, 06 Apr 2026 11:41:19 -0400</pubDate>
      
      <guid>https://secopsmonkey.com/post/2026-04-06-tool-library-announcement/</guid>
      
        <description>&lt;p&gt;During a recent conversation someone pointed out that my &lt;a href=&#34;https://github.com/packs&#34;&gt;Github profile&lt;/a&gt; is somewhat dusty. Afterwards I took a public
view and dang, they&amp;rsquo;re absolutely right. Nearly all of my recent work has been restricted to private repos or owned by my
employers.&lt;/p&gt;
&lt;p&gt;Let&amp;rsquo;s fix that, shall we? Starting today I plan to go through all of the various scripts and applications I&amp;rsquo;ve written over
the years. For each script I&amp;rsquo;ll tidy them up enough for a public release and publish a blurb about each one.&lt;/p&gt;
&lt;p&gt;To kick off the process I&amp;rsquo;ve released &lt;a href=&#34;https://github.com/packs/entra_management/tree/main/Update-UserAttributes&#34;&gt;Update-UserAttributes&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&#34;background&#34;&gt;Background&lt;/h2&gt;
&lt;p&gt;Like any good Internet recipe, here&amp;rsquo;s some context-flavored reminiscing.&lt;/p&gt;
&lt;p&gt;It was the halcyon days of late 2024 and I had recently taken over as technical owner of our Microsoft 365 environment. One
of my primary goals was to cleanup and modernize our technical practices. Previously, all Microsoft 365 administration had been
manual. The result was a lot of cruft, inconsistent data, and inefficient processes. One of the first steps of an overall
Identity Management Improvement project was to make sure our user data was accurate and trustworthy. This meant defining an
authoritative data source for user information and ensuring that the downstream systems, in this case Entra, were kept up to date.&lt;/p&gt;
&lt;p&gt;Since Human Resources used Paylocity as their HRIS, this meant setting up the integration for user attribute updates. The integration
supports automatic user creation, attribute updates, and account disabling. One hiccup, Paylocity is a purely event driven system.
Meaning, it had no way of correcting Entra directory information without individually modifying each and every person in Paylocity.&lt;/p&gt;
&lt;p&gt;Thus &lt;code&gt;Update-UserAttributes&lt;/code&gt; was born.&lt;/p&gt;
&lt;h2 id=&#34;the-intent&#34;&gt;The Intent&lt;/h2&gt;
&lt;p&gt;This script, &lt;code&gt;Update-UserAttributes.ps1&lt;/code&gt;, was designed to make bulk modifications to Entra user objects. In my use case, it was to
take an export from Paylocity and perform a reconciliation of directory information (Job Titles, Managers, Departments, etc). In
the general case, I wanted it to be useful as a generic user information management tool. Not every attribute is owned by HR, and this
would give us the opportunity to manage them as needed.&lt;/p&gt;
&lt;h2 id=&#34;usage&#34;&gt;Usage&lt;/h2&gt;
&lt;p&gt;The best use case is bulk-updating multiple attributes across multiple users simultaneously. To do this, create a CSV file using the
following format. Something important to note is that the header name must match the property name used by the &lt;a href=&#34;https://learn.microsoft.com/en-us/powershell/module/microsoft.graph.users/update-mguser?view=graph-powershell-1.0&#34;&gt;&lt;code&gt;Update-MgUser&lt;/code&gt;&lt;/a&gt; cmdlet.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;div class=&#34;chroma&#34;&gt;
&lt;table class=&#34;lntable&#34;&gt;&lt;tr&gt;&lt;td class=&#34;lntd&#34;&gt;
&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code&gt;&lt;span class=&#34;lnt&#34;&gt;1
&lt;/span&gt;&lt;span class=&#34;lnt&#34;&gt;2
&lt;/span&gt;&lt;span class=&#34;lnt&#34;&gt;3
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;
&lt;td class=&#34;lntd&#34;&gt;
&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-csv&#34; data-lang=&#34;csv&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;s&#34;&gt;UserId&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;s&#34;&gt;jobTitle&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;s&#34;&gt;department&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;s&#34;&gt;Manager&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;s&#34;&gt;user1@domain.com&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;s&#34;&gt;Sales Director&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;s&#34;&gt;Sales&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;s&#34;&gt;manager@domain.com&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;s&#34;&gt;user2@domain.com&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;s&#34;&gt;Engineer&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;s&#34;&gt;Development&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;s&#34;&gt;manager@domain.com&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Once you create the CSV you can make the upstream changes by passing it on the command line&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-powershell&#34; data-lang=&#34;powershell&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;p&#34;&gt;.\&lt;/span&gt;&lt;span class=&#34;nb&#34;&gt;Update-UserAttributes&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;py&#34;&gt;ps1&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;-CSVFilePath&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;.\&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;userlist&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;csv&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;More details can be found in the script&amp;rsquo;s README and Help output.&lt;/p&gt;
&lt;h2 id=&#34;caveats&#34;&gt;Caveats&lt;/h2&gt;
&lt;p&gt;There&amp;rsquo;s definitely room for improvement. For example:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Add &lt;code&gt;-WhatIf&lt;/code&gt; support&lt;/li&gt;
&lt;li&gt;Move Error reporting to &lt;code&gt;Write-Error&lt;/code&gt; and add gobs more&lt;/li&gt;
&lt;li&gt;Actually implement exception handling&lt;/li&gt;
&lt;li&gt;Implement output logging sufficient to be auditable&lt;/li&gt;
&lt;li&gt;Validate CSV input&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Please don&amp;rsquo;t be shy with your feature and/or pull requests. My main reason for publishing is to make it available in the hope
that someone else finds it useful.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>auditd By Example - Monitoring Process Execution</title>
      <link>https://secopsmonkey.com/post/2018-07-24-monitoring-process-execution-with-auditd/</link>
      <pubDate>Tue, 24 Jul 2018 12:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2018-07-24-monitoring-process-execution-with-auditd/</guid>
      
        <description>&lt;p&gt;A fellow &lt;a href=&#34;http://www.brakeingsecurity.com/&#34;&gt;Brakeing Down Security&lt;/a&gt; slacker, &lt;a href=&#34;https://twitter.com/clintmilner&#34;&gt;Ceafin&lt;/a&gt; asked a fun question question to the group at large.&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2018-07-24-monitoring-process-execution-with-auditd/Ceafin_Process_Question.png&#34;
    alt=&#34;Process Monitoring Question&#34;&gt;
&lt;/figure&gt;

&lt;p&gt;He further went on to explain that he was investigating a potentially compromised system and didn&amp;rsquo;t trust the standard tools
or logs.&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2018-07-24-monitoring-process-execution-with-auditd/Ceafin_Additional_Information.png&#34;
    alt=&#34;Process Monitoring Info&#34;&gt;
&lt;/figure&gt;

&lt;p&gt;While there were a few good answers that other users brought up my first instinct was good ole fashioned &lt;code&gt;auditd&lt;/code&gt;. Since the
auditing subsystem operates as a series of kernel hooks this &lt;em&gt;should&lt;/em&gt;&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a href=&#34;#fn:1&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt; give low enough level insight to bypass most simple
evasion.&lt;/p&gt;
&lt;p&gt;The high level workflow here would be to let &lt;code&gt;auditd&lt;/code&gt; log process execution events, then once logged we can review the events
at our leisure. Given the IR style setup we have going on we have to be a little suspicious of any information we generate or
view. If log forwarding were configured then we&amp;rsquo;d be in a little better place, however we should allow for the possibility of
kernel level injection that interferes with our data collection. In my opinion interference at this level is unlikely, however
I don&amp;rsquo;t do DFIR much so I could easily be wrong.&lt;/p&gt;
&lt;p&gt;In the end we&amp;rsquo;re looking at a surprisingly simple ruleset. Let&amp;rsquo;s create a new rules snippet called &lt;code&gt;/etc/audit/rules.d/10-procmon.rules&lt;/code&gt;
and add the lines&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;-a exit,always -F &lt;span class=&#34;nv&#34;&gt;arch&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;b64 -S execve -k procmon
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;-a exit,always -F &lt;span class=&#34;nv&#34;&gt;arch&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;b32 -S execve -k procmon&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;By using the &lt;code&gt;-a&lt;/code&gt; we are creating a syscall rule. This rule should &lt;code&gt;always&lt;/code&gt; create an event when a system call &lt;code&gt;exit&lt;/code&gt;s&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a href=&#34;#fn:2&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;. Instead
of triggering an event on every system call we instead only want to log when a program call &lt;code&gt;execve(2)&lt;/code&gt;. Lastly let&amp;rsquo;s append the
key value &amp;ldquo;procmon&amp;rdquo; to each log for easier searching. Since 64-bit kernels will sometimes have a different syscall entry path for
each architecture, &lt;code&gt;auditd&lt;/code&gt; wants us to explicitly define which architecture we want to monitor system calls for. If this was a 32-bit
system I&amp;rsquo;d simply remove one of the lines and strip out the &lt;code&gt;arch&lt;/code&gt; filter. Lastly, we use the &lt;code&gt;-k&lt;/code&gt; argument to tag each log entry
with a specific keyword for easier searching.&lt;/p&gt;
&lt;p&gt;Once we&amp;rsquo;ve added the rules (either manually with &lt;code&gt;auditctl&lt;/code&gt; or by restarting the &lt;code&gt;auditd&lt;/code&gt; service) we can check up on our work by
reviewing the log file. As an example I decided to simply ran the command &lt;code&gt;cat 10-procmon.rules&lt;/code&gt; to dump the rules out to the
screen. Looking through &lt;code&gt;/var/log/auditd.log&lt;/code&gt; for our logs we can see the following.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nv&#34;&gt;type&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;SYSCALL &lt;span class=&#34;nv&#34;&gt;msg&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;audit&lt;span class=&#34;o&#34;&gt;(&lt;/span&gt;1532489108.216:3721&lt;span class=&#34;o&#34;&gt;)&lt;/span&gt;: &lt;span class=&#34;nv&#34;&gt;arch&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;c000003e &lt;span class=&#34;nv&#34;&gt;syscall&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;59&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;success&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;yes &lt;span class=&#34;nv&#34;&gt;exit&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;a0&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;16169e0 &lt;span class=&#34;nv&#34;&gt;a1&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;16116f0 &lt;span class=&#34;nv&#34;&gt;a2&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;161ab60 &lt;span class=&#34;nv&#34;&gt;a3&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;7ffd940300a0 &lt;span class=&#34;nv&#34;&gt;items&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;2&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;ppid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;10627&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;pid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;20240&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;auid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;1000&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;uid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;gid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;euid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;suid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;fsuid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;egid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;sgid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;fsgid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;tty&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;pts1 &lt;span class=&#34;nv&#34;&gt;ses&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;3&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;comm&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;cat&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;exe&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;/usr/bin/cat&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;subj&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 &lt;span class=&#34;nv&#34;&gt;key&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;procmon&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nv&#34;&gt;type&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;EXECVE &lt;span class=&#34;nv&#34;&gt;msg&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;audit&lt;span class=&#34;o&#34;&gt;(&lt;/span&gt;1532489108.216:3721&lt;span class=&#34;o&#34;&gt;)&lt;/span&gt;: &lt;span class=&#34;nv&#34;&gt;argc&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;2&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;a0&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;cat&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;a1&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;10-procmon.rules&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nv&#34;&gt;type&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;SYSCALL &lt;span class=&#34;nv&#34;&gt;msg&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;audit&lt;span class=&#34;o&#34;&gt;(&lt;/span&gt;1532489109.283:3722&lt;span class=&#34;o&#34;&gt;)&lt;/span&gt;: &lt;span class=&#34;nv&#34;&gt;arch&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;c000003e &lt;span class=&#34;nv&#34;&gt;syscall&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;59&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;success&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;yes &lt;span class=&#34;nv&#34;&gt;exit&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;a0&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;1f14e30 &lt;span class=&#34;nv&#34;&gt;a1&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;1f1a110 &lt;span class=&#34;nv&#34;&gt;a2&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;1f21d20 &lt;span class=&#34;nv&#34;&gt;a3&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;7ffc85e6eee0 &lt;span class=&#34;nv&#34;&gt;items&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;2&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;ppid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;10382&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;pid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;20241&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;auid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;1000&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;uid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;gid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;euid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;suid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;fsuid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;egid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;sgid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;fsgid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;tty&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;pts0 &lt;span class=&#34;nv&#34;&gt;ses&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;1&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;comm&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;grep&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;exe&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;/usr/bin/grep&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;subj&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 &lt;span class=&#34;nv&#34;&gt;key&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;procmon&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nv&#34;&gt;type&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;EXECVE &lt;span class=&#34;nv&#34;&gt;msg&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;audit&lt;span class=&#34;o&#34;&gt;(&lt;/span&gt;1532489109.283:3722&lt;span class=&#34;o&#34;&gt;)&lt;/span&gt;: &lt;span class=&#34;nv&#34;&gt;argc&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;4&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;a0&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;grep&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;a1&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;--color=auto&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;a2&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;procmon&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;a3&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;/var/log/audit/audit.log&amp;#34;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;As discussed in the &lt;a href=&#34;http://secopsmonkey.com/a-brief-introduction-to-auditd.html&#34;&gt;introductory blog post&lt;/a&gt; the logs are&amp;hellip;somewhat verbose. The key points are to notice the format of the
entries. First we have a type SYSCALL with the &amp;ldquo;exe&amp;rdquo; entry of &lt;code&gt;/usr/bin/cat&lt;/code&gt; followed immediately by a log of type EXECVE. Normally
writing an event on a syscall results in a single entry that contains the process name and not much else. However, the &lt;code&gt;execve&lt;/code&gt;
call is different. Monitoring for this particular system call automatically triggers an additional log of type EXECVE. This log
type also includes the string values of the function arguments as opposed to the value of the pointer (see the a0 discussion under
Fields in the &lt;a href=&#34;https://linux.die.net/man/8/auditctl&#34;&gt;man page&lt;/a&gt;). We can see that the first entry says I ran the command &lt;code&gt;cat 10-procmon.rules&lt;/code&gt;. As a fun aside we
can also see in the second log entry where I searched for this information using the command &lt;code&gt;grep procmon /var/log/audit/audit.log&lt;/code&gt;.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;For some definitions of &amp;ldquo;should&amp;rdquo;.&amp;#160;&lt;a href=&#34;#fnref:1&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;The use of &lt;code&gt;exit&lt;/code&gt; here may be a little confusing. In auditd parlance this means when the system call execution is completed but before any data is returned to the calling application. A high level architecture diagram of the workflow can be found in the &lt;a href=&#34;https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/chap-system_auditing#sec-audit_system_architecture&#34;&gt;RedHat documentation&lt;/a&gt;.&amp;#160;&lt;a href=&#34;#fnref:2&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>Tracking Down Silent SELinux Denies</title>
      <link>https://secopsmonkey.com/post/2018-01-22-tracking-down-silent-selinux-denies/</link>
      <pubDate>Mon, 22 Jan 2018 12:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2018-01-22-tracking-down-silent-selinux-denies/</guid>
      
        <description>&lt;h3 id=&#34;the-setup&#34;&gt;The Setup&lt;/h3&gt;
&lt;p&gt;Let&amp;rsquo;s paint a picture of a fun scenario. In my home lab I run a mix of CentOS and Windows. Windows
for Active Directory and those Windows specific apps, CentOS for most everything else. For CentOS
systems I use &lt;a href=&#34;https://spacewalkproject.github.io/&#34;&gt;Spacewalk&lt;/a&gt; as an inventory and patch distribution system. While Spacewalk is pretty
dang heavy for such a simple workflow it at least keeps me tuned into what is likely to be found in
production environments. Rather than using local accounts in Spacewalk I configured it to use PAM,
which in turn uses &lt;code&gt;pam_krb5&lt;/code&gt; connected to my lab&amp;rsquo;s Windows domain.&lt;/p&gt;
&lt;p&gt;One day during sign-in I noticed the &lt;code&gt;pam_krb5&lt;/code&gt; connection had broken. This being a lab setup I don&amp;rsquo;t
sign into the back-end services very often, which means I have no idea &lt;em&gt;when&lt;/em&gt; the service stopped
working. The web interface throws a generic &amp;ldquo;Invalid Username/Password&amp;rdquo; error and the system throws these
two messages:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;Dec  &lt;span class=&#34;m&#34;&gt;6&lt;/span&gt; 19:22:27 node1.localdomain server&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;1186&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt;: 2017-12-06 19:22:27,669 &lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;ajp-bio-0:0:0:0:0:0:0:1-8009-exec-7&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt; WARN  com.redhat.rhn.domain.user.legacy.UserImpl - PAM login &lt;span class=&#34;k&#34;&gt;for&lt;/span&gt; user User spack &lt;span class=&#34;o&#34;&gt;(&lt;/span&gt;id 2, org_id 1&lt;span class=&#34;o&#34;&gt;)&lt;/span&gt; failed with error System error.
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;Dec  &lt;span class=&#34;m&#34;&gt;6&lt;/span&gt; 19:22:29 node1.localdomain server&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;1186&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt;: 2017-12-06 19:22:29,670 &lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;ajp-bio-0:0:0:0:0:0:0:1-8009-exec-7&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt; INFO  com.redhat.rhn.frontend.action.LoginAction - LOCAL AUTH FAILURE: &lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;spack&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;I&amp;rsquo;ve been dealing with SELinux long enough that it is the first place I check, but alas no AVC
messages to be found in the audit log. After some period of time bumbling around, and verifying
&lt;code&gt;pam_krb5&lt;/code&gt; works for other services, I decide to disable SELinux and see what happens. Lo and behold,
I can sign-in just fine. Clearly there is an AVC event but it is not getting logged.&lt;/p&gt;
&lt;h3 id=&#34;the-diagnosis&#34;&gt;The Diagnosis&lt;/h3&gt;
&lt;p&gt;Further reading indicates that it is, in fact, perfectly normal and expected for denies to be silent.
There are a number of reasons this might be the case, but there is an expectation that &lt;em&gt;most&lt;/em&gt; denies
are cosmetic noise and should not be seen. Since the decision to hide a denial is determined by the
policy author we may occasionally find scenarios where a denial is, in fact, not cosmetic and is
instead a real problem.&lt;/p&gt;
&lt;p&gt;The critical tool here is &lt;code&gt;seinfo&lt;/code&gt;, which is shipped as part of the &lt;code&gt;setools-console&lt;/code&gt; package. It
spits out a bunch of nice statistical information about the running configuration, but what&amp;rsquo;s really
relevant here can be found with&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;root@node1 ~&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;c1&#34;&gt;# seinfo --stats | grep audit&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;   Auditallow:        &lt;span class=&#34;m&#34;&gt;155&lt;/span&gt;    Dontaudit:        &lt;span class=&#34;m&#34;&gt;8855&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;This is showing that of all the loaded policies we have 155 explicit allows configured and a grand total
of 8855 rules set to block but not log. That&amp;rsquo;s quite a lot of denies we normally don&amp;rsquo;t get to see, so
presumably one of those is causing our problem.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;dontaudit&lt;/code&gt; statement is a keyword that disables logging on a particular rule. The easiest approach
for us right now is to disable the &lt;code&gt;dontaudit&lt;/code&gt; keyword. By doing so it won&amp;rsquo;t take affect and, more
importantly, we won&amp;rsquo;t be required to modify each shipped policy.  We can do this by running:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;semodule --disable_dontaudit --build&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;After a few minutes the rules will be recompiled. Now if we go back to the webpage and try signing in
again we get receive the failed sign-in error, as expected, but this time there&amp;rsquo;s an audit entry:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nv&#34;&gt;type&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;SYSCALL &lt;span class=&#34;nv&#34;&gt;msg&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;audit&lt;span class=&#34;o&#34;&gt;(&lt;/span&gt;1513616321.280:10659&lt;span class=&#34;o&#34;&gt;)&lt;/span&gt;: &lt;span class=&#34;nv&#34;&gt;arch&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;c000003e &lt;span class=&#34;nv&#34;&gt;syscall&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;44&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;success&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;no &lt;span class=&#34;nv&#34;&gt;exit&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;-13 &lt;span class=&#34;nv&#34;&gt;a0&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;41&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;a1&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;7f1fe06b8ef0 &lt;span class=&#34;nv&#34;&gt;a2&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;c0 &lt;span class=&#34;nv&#34;&gt;a3&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;items&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;ppid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;1&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;pid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;1186&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;auid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;4294967295&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;uid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;91&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;gid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;91&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;euid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;91&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;suid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;91&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;fsuid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;91&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;egid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;91&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;sgid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;91&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;fsgid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;91&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;tty&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=(&lt;/span&gt;none&lt;span class=&#34;o&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;ses&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;4294967295&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;comm&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;java&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;exe&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-1.b12.el7_4.x86_64/jre/bin/java&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;subj&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;system_u:system_r:tomcat_t:s0 &lt;span class=&#34;nv&#34;&gt;key&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=(&lt;/span&gt;null&lt;span class=&#34;o&#34;&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nv&#34;&gt;type&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;AVC &lt;span class=&#34;nv&#34;&gt;msg&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;audit&lt;span class=&#34;o&#34;&gt;(&lt;/span&gt;1513616321.280:10659&lt;span class=&#34;o&#34;&gt;)&lt;/span&gt;: avc:  denied  &lt;span class=&#34;o&#34;&gt;{&lt;/span&gt; write &lt;span class=&#34;o&#34;&gt;}&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;for&lt;/span&gt;  &lt;span class=&#34;nv&#34;&gt;pid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;1186&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;comm&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;java&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;scontext&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;system_u:system_r:tomcat_t:s0 &lt;span class=&#34;nv&#34;&gt;tcontext&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;system_u:system_r:tomcat_t:s0 &lt;span class=&#34;nv&#34;&gt;tclass&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;netlink_audit_socket&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;It looks like the existing policies likely didn&amp;rsquo;t handle an upgrade very well.&lt;/p&gt;
&lt;h3 id=&#34;the-repair&#34;&gt;The Repair&lt;/h3&gt;
&lt;p&gt;In addition to the normal search and modification tools there&amp;rsquo;s a great utility called &lt;code&gt;audit2allow&lt;/code&gt; in
&lt;code&gt;policycoreutils-python&lt;/code&gt; that can be used to automagically generate policy files from audit logs. Using
the excellent &lt;a href=&#34;https://wiki.centos.org/HowTos/SELinux#head-faa96b3fdd922004cdb988c1989e56191c257c01&#34;&gt;SELinux HowTo&lt;/a&gt; we can build a local policy and push it out. In my case there were two
permits we needed to add. This is, of course, just a policy snippet of the specific changes I made. Your
new policy may look very different.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;module spacewalklocalpam 1.0&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;require &lt;span class=&#34;o&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;        &lt;span class=&#34;nb&#34;&gt;type&lt;/span&gt; tomcat_t&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;        class netlink_audit_socket &lt;span class=&#34;o&#34;&gt;{&lt;/span&gt; write &lt;span class=&#34;nb&#34;&gt;read&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;}&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;o&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;allow tomcat_t self:netlink_audit_socket &lt;span class=&#34;o&#34;&gt;{&lt;/span&gt; write &lt;span class=&#34;nb&#34;&gt;read&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;}&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Once loaded and tested make sure to re-enable the &lt;code&gt;dontaudit&lt;/code&gt; option. (Unless you don&amp;rsquo;t want to, that&amp;rsquo;s your log volume not mine) by
running&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;semodule --build&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>Deploying LibreNMS With SELinux</title>
      <link>https://secopsmonkey.com/post/2017-12-11-deploying-librenms-with-selinux/</link>
      <pubDate>Mon, 11 Dec 2017 12:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2017-12-11-deploying-librenms-with-selinux/</guid>
      
        <description>&lt;p&gt;During an infrastructure upgrade we decided to migrate from Observium to
LibreNMS for our time-series graphs. That system is one of the last vestiges
of the Ubuntu standardization and will be one of the biggest wins. Since it
was one of the first forays into Linux for the team some of the build decisions
were less than enterprise; for instance this system runs Ubuntu 12.04 Desktop.
Part of the replacement includes migrating to CentOS and, relevant to this
post, enabling SELinux by default in enforcing mode.&lt;/p&gt;
&lt;p&gt;The LibreNMS project does a lot of things right, but one thing that&amp;rsquo;s missing
is a well developed SELinux policy out of the box. For the purposes of building
out this policy we made the following assumptions:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The application is installed in &lt;code&gt;/opt/librenms&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;We are using Apache httpd as the web server&lt;/li&gt;
&lt;li&gt;We are using MariaDB/MySQL as the database engine&lt;/li&gt;
&lt;li&gt;We use an LDAP/AD back-end for user authentication&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Building the policy itself was fairly easy. All we needed to explicitly call out was
that the httpd process needed to make SNMP and LDAP connections. The LDAP connection
is for authentication, and SNMP for&amp;hellip;well&amp;hellip;this &lt;em&gt;is&lt;/em&gt; a monitoring server&amp;hellip;.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;module librenms 1.0&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;require &lt;span class=&#34;o&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;nb&#34;&gt;type&lt;/span&gt; ldap_port_t&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;nb&#34;&gt;type&lt;/span&gt; snmp_port_t&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;nb&#34;&gt;type&lt;/span&gt; httpd_t&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    class tcp_socket name_connect&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;o&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;#============= httpd_t ==============&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;#!!!! This avc can be allowed using one of the these booleans:&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;#     httpd_can_connect_ldap, nis_enabled, authlogin_nsswitch_use_ldap, httpd_can_network_connect&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;allow httpd_t ldap_port_t:tcp_socket name_connect&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;#!!!! This avc can be allowed using one of the these booleans:&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;#     nis_enabled, httpd_can_network_connect&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;allow httpd_t snmp_port_t:tcp_socket name_connect&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Next came the more interesting part. By default the LibreNMS application itself logs to
&lt;code&gt;/opt/librenms/logs&lt;/code&gt;, however this is well outside of the web root and would be blocked.
To remove the bock we need to set the proper security contexts on the log directory. In
order to make it a permanent change we need to create a file context entry using the
command:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;semanage fcontext -a -t httpd_sys_rw_content_t &lt;span class=&#34;s1&#34;&gt;&amp;#39;/opt/librenms/logs(/.*)?&amp;#39;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;The last step is fixing the cleanup process. Without making any other changes the application
can create and update rrd logs for all of the devices, however decommissioning causes an issue.
When we try to delete the rrd directory as part of a server decommissioning the deletion fails
with an access denied message. We also need to define the file contexts such that the httpd
process can delete the files.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;semanage fcontext -a -t httpd_sys_rw_content_t &lt;span class=&#34;s1&#34;&gt;&amp;#39;/opt/librenms/rrd(/.*)?&amp;#39;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Finally we can go actually set the contexts on all those files:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;restorecon -R -v /opt/librenms/rrd /opt/librenms/logs&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;At this point we were able to add, delete, and update hosts; poll and report on polling; hell
even add some custom MIBs for graphs that aren&amp;rsquo;t shipped out of the box.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>auditd By Example - Tracking File Changes</title>
      <link>https://secopsmonkey.com/post/2015-08-07-auditd-by-example-tracking-file-changes/</link>
      <pubDate>Fri, 07 Aug 2015 12:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2015-08-07-auditd-by-example-tracking-file-changes/</guid>
      
        <description>&lt;p&gt;ServerFault user &lt;a href=&#34;http://serverfault.com/users/13325/ewwhite&#34;&gt;ewwhite&lt;/a&gt; &lt;a href=&#34;http://meta.security.stackexchange.com/a/1411/618&#34;&gt;describes&lt;/a&gt; a rather interesting situation regarding application distribution wherein code must be compiled in
production. In short he wants to keep track of changes to a specific directory path and send alerts via email.&lt;/p&gt;
&lt;p&gt;Let&amp;rsquo;s assume that there already exists some basic form of &lt;code&gt;auditd&lt;/code&gt; policy in play, so we&amp;rsquo;ll be building out a snippet to be inserted into your
existing &lt;code&gt;/etc/audit/audit.rules&lt;/code&gt;. Ed was sparse on some of the specifics related to the application, understandably so, so let&amp;rsquo;s make some
additional assumptions. Let&amp;rsquo;s assume that the source code directory in question is &lt;code&gt;/opt/application/src&lt;/code&gt; and that all binaries are installed into
&lt;code&gt;/opt/application/bin&lt;/code&gt;. With those assumptions in place let&amp;rsquo;s add these rules.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;-w /opt/application/src -p w -k app-policy
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;-w /opt/application/bin -p wa -k app-policy&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;I&amp;rsquo;ve decided to process each directory, source and binaries, separately. The commonality between the two are the &lt;code&gt;-w&lt;/code&gt; and the &lt;code&gt;-k&lt;/code&gt; options. The &lt;code&gt;-w&lt;/code&gt;
option says to watch that directory recursively. The &lt;code&gt;-k&lt;/code&gt; option says to add the text &lt;code&gt;app-policy&lt;/code&gt;, called a key, to the output log message, this is
just to make log reviews easier. The &lt;code&gt;-p&lt;/code&gt; option is actually where the magic happens, and is the real reason to separate these two rules out.&lt;/p&gt;
&lt;p&gt;As we discussed in the &lt;a href=&#34;http://secopsmonkey.com/a-brief-introduction-to-auditd.html&#34;&gt;introductory post&lt;/a&gt; &lt;code&gt;-p w&lt;/code&gt; instructs the kernel to log on file writes. One would assume this is accomplished by attaching
to the POSIX system call &lt;code&gt;write()&lt;/code&gt;, I certainly did. In truth that syscall gets called quite a lot when files are actually saved. So as to not
overwhelm the logging system auditd instead attaches to the &lt;code&gt;open()&lt;/code&gt; system call. By using the (w)rite argument we look for any instance of &lt;code&gt;open()&lt;/code&gt;
that uses the &lt;code&gt;O_WRONLY&lt;/code&gt; or &lt;code&gt;O_RDWR&lt;/code&gt; flags. It&amp;rsquo;s worth noting that this does not mean a file was actually modified, only that it was opened in such
a way that would allow for modification. For example, if a user opened &lt;code&gt;/opt/application/src/app.h&lt;/code&gt; in a text editor a log would be generated, however
if it was written to the terminal using &lt;code&gt;cat&lt;/code&gt; or read using &lt;code&gt;less&lt;/code&gt; then no log would be generated. This is pretty important to remember as many people
will read a file using a text editor and simply exit without saving changes (hopefully).&lt;/p&gt;
&lt;p&gt;We also want to watch for file writes in the binary directory except here we would expect them to be more reliable. It would be rather unusual, but
not out of the question, for someone to attempt to use a text editor to open an executable. In addition we added the (a)ttribute option. This will
alert us if any of the ownership or access permissions change, most importantly if a file is changed to be executable or the ownership is changed.
This will not catch SELinux context changes but since SELinux uses the auditd logging engine then those changes will still be logged and saved into
the same log file.&lt;/p&gt;
&lt;p&gt;Now that we have the rules constructed we can move on to the alerting. Ed wanted the events to be emailed out. This is actually quite a bit more
complicated. By default &lt;code&gt;auditd&lt;/code&gt; uses its own built in logging engine instead of relying on something external like &lt;code&gt;syslogd&lt;/code&gt; or `rsyslog. By not
relying on an external logger it is better able to withstand misconfigurations. However, it also means that making modifications can be trickier
because the custom engine means mind-share is significantly smaller and will require yet another expertise requirement on your team.&lt;/p&gt;
&lt;p&gt;There does exist a subsystem called &lt;code&gt;audispd&lt;/code&gt; that acts as a log multiplexer. There are a number of output plugins available, such as syslog, UNIX
socket, prelude IDS, etc. None of them really do what Ed wants, so I think our best bet would run reports. Auditd is, after all, an auditing tool and
&lt;code&gt;not&lt;/code&gt; an enforcement tool. So let&amp;rsquo;s look at something a little different.&lt;/p&gt;
&lt;p&gt;Remember how we tacked on &lt;code&gt;-k app-policy&lt;/code&gt; to those rules above? Now we get to the why. Let&amp;rsquo;s try running the command:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;aureport -k -ts yesterday 00:00:00 -te yesterday 23:59:59&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;We should now see a list of all of the logs that contain any keys and occurred yesterday. Let&amp;rsquo;s look at a concrete example of me editing a file in
that directory and the subsequent logs.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;root@ node1:~&amp;gt; mkdir -p /opt/application/src
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;root@ node1:~&amp;gt; vim /opt/application/src/app.h
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;root@ node1:~&amp;gt; aureport -k
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;Key &lt;span class=&#34;nv&#34;&gt;Report&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;o&#34;&gt;===============================================&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# date time key success exe auid event&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;o&#34;&gt;===============================================&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;1. 09/24/2013 11:41:29 app-policy yes /usr/bin/vim &lt;span class=&#34;m&#34;&gt;1000&lt;/span&gt; &lt;span class=&#34;m&#34;&gt;13446&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;2. 09/24/2013 11:41:29 app-policy yes /usr/bin/vim &lt;span class=&#34;m&#34;&gt;1000&lt;/span&gt; &lt;span class=&#34;m&#34;&gt;13445&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;3. 09/24/2013 11:41:29 app-policy yes /usr/bin/vim &lt;span class=&#34;m&#34;&gt;1000&lt;/span&gt; &lt;span class=&#34;m&#34;&gt;13447&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;4. 09/24/2013 11:41:29 app-policy yes /usr/bin/vim &lt;span class=&#34;m&#34;&gt;1000&lt;/span&gt; &lt;span class=&#34;m&#34;&gt;13448&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;5. 09/24/2013 11:41:29 app-policy yes /usr/bin/vim &lt;span class=&#34;m&#34;&gt;1000&lt;/span&gt; &lt;span class=&#34;m&#34;&gt;13449&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;6. 09/24/2013 11:41:35 app-policy yes /usr/bin/vim &lt;span class=&#34;m&#34;&gt;1000&lt;/span&gt; &lt;span class=&#34;m&#34;&gt;13451&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;7. 09/24/2013 11:41:35 app-policy yes /usr/bin/vim &lt;span class=&#34;m&#34;&gt;1000&lt;/span&gt; &lt;span class=&#34;m&#34;&gt;13450&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;The report tells us that at 11:41:29 on September the 24th a user ran the command &lt;code&gt;/usr/bin/vim&lt;/code&gt; and triggered a rule labeled &lt;code&gt;app-policy&lt;/code&gt;. It&amp;rsquo;s all
good so far, but not very detailed. The last two fields, however, are quite useful. The first, 1000, is the UID of my personal account. That &lt;em&gt;is&lt;/em&gt;
important because notice I was actually running as root. Since I had originally used &lt;code&gt;sudo -i&lt;/code&gt; to gain a root shell my original UID was still
preserved, this is good! The last field is a unique event ID generated by auditd. Let&amp;rsquo;s look at that first event, numbered 13446.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;root@ node1:~&amp;gt; grep :13446 /var/log/audit/audit.log
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nv&#34;&gt;type&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;SYSCALL &lt;span class=&#34;nv&#34;&gt;msg&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;audit&lt;span class=&#34;o&#34;&gt;(&lt;/span&gt;1380037289.364:13446&lt;span class=&#34;o&#34;&gt;)&lt;/span&gt;: &lt;span class=&#34;nv&#34;&gt;arch&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;c000003e &lt;span class=&#34;nv&#34;&gt;syscall&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;2&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;success&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;yes &lt;span class=&#34;nv&#34;&gt;exit&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;4&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;a0&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;bffa20 &lt;span class=&#34;nv&#34;&gt;a1&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;c2 &lt;span class=&#34;nv&#34;&gt;a2&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;180&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;a3&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;items&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;2&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;ppid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;21950&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;pid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;22277&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;auid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;1000&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;uid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;gid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;euid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;suid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;fsuid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;egid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;sgid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;fsgid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;ses&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;1226&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;tty&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;pts0 &lt;span class=&#34;nv&#34;&gt;comm&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;vim&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;exe&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;/usr/bin/vim&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;subj&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 &lt;span class=&#34;nv&#34;&gt;key&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;app-policy&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nv&#34;&gt;type&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;CWD &lt;span class=&#34;nv&#34;&gt;msg&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;audit&lt;span class=&#34;o&#34;&gt;(&lt;/span&gt;1380037289.364:13446&lt;span class=&#34;o&#34;&gt;)&lt;/span&gt;:  &lt;span class=&#34;nv&#34;&gt;cwd&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;/root&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nv&#34;&gt;type&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;PATH &lt;span class=&#34;nv&#34;&gt;msg&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;audit&lt;span class=&#34;o&#34;&gt;(&lt;/span&gt;1380037289.364:13446&lt;span class=&#34;o&#34;&gt;)&lt;/span&gt;: &lt;span class=&#34;nv&#34;&gt;item&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;name&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;/opt/application/src/&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;inode&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;2747242&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;dev&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;fd:01 &lt;span class=&#34;nv&#34;&gt;mode&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;040755&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;ouid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;ogid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;rdev&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;00:00 &lt;span class=&#34;nv&#34;&gt;obj&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;unconfined_u:object_r:usr_t:s0
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nv&#34;&gt;type&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;PATH &lt;span class=&#34;nv&#34;&gt;msg&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;audit&lt;span class=&#34;o&#34;&gt;(&lt;/span&gt;1380037289.364:13446&lt;span class=&#34;o&#34;&gt;)&lt;/span&gt;: &lt;span class=&#34;nv&#34;&gt;item&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;1&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;name&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;/opt/application/src/.app.h.swx&amp;#34;&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;inode&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;2747244&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;dev&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;fd:01 &lt;span class=&#34;nv&#34;&gt;mode&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0100600&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;ouid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;ogid&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;0&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;rdev&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;00:00 &lt;span class=&#34;nv&#34;&gt;obj&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;unconfined_u:object_r:usr_t:s0&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;This is what we mean when we say audit logs are verbose. In the &lt;a href=&#34;http://secopsmonkey.com/a-brief-introduction-to-auditd.html&#34;&gt;introductory blog post&lt;/a&gt; we discussed some of those fields so I&amp;rsquo;ll save us the
pain of going over it again. What we can see, however, is that the user with uid 1000 (see auid=1000) ran the command vim as root (see euid=0) and
that the command resulted in a change to both &lt;code&gt;/opt/application/src/&lt;/code&gt; and &lt;code&gt;/opt/application/src/.app.h.swx&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;What we should be able to see here is that the report generated by aureport doesn&amp;rsquo;t contain everything we need to see what happened, but it does tell us
&lt;em&gt;something&lt;/em&gt; happened and gives us the necessary data to find the information. In an ideal world you would have some kind of log aggregation system,
like Splunk or Graylog or a SIEM, and send the raw logs there. That system would then have all the alerting functionality built in to alert an admin
to the potential policy violation. However, we don&amp;rsquo;t live in a perfect world and Ed&amp;rsquo;s request for email alerts implies he doesn&amp;rsquo;t have access to such
a system. What I would do is set up a daily cron job to run that report for the previous day. Every morning the log reviewer can check their mailbox
and see if any of those files changed when they weren&amp;rsquo;t supposed to. If daily isn&amp;rsquo;t reactive enough then we can simply change the values passed to
&lt;code&gt;-ts&lt;/code&gt; and &lt;code&gt;-te&lt;/code&gt; and run the job more frequently.&lt;/p&gt;
&lt;p&gt;Pulling it all together we should have something that looks like this.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;#/etc/audit/audit.rules&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# This file contains the auditctl rules that are loaded&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# whenever the audit daemon is started via the initscripts.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# The rules are simply the parameters that would be passed&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# to auditctl.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# First rule - delete all&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;-D
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Increase the buffers to survive stress events.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Make this bigger for busy systems&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;-b &lt;span class=&#34;m&#34;&gt;320&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Feel free to add below this line. See auditctl man page&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;-w /opt/application/src -p w -k app-policy
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;-w /opt/application/bin -p wa -k app-policy&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;#/etc/cron.d/audit-report&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nv&#34;&gt;MAILTO&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;ewwhite@example.com
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;m&#34;&gt;1&lt;/span&gt; &lt;span class=&#34;m&#34;&gt;0&lt;/span&gt;   * * *     root  /sbin/aureport -k -ts yesterday 00:00:00 -te yesterday 23:59:59&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>Upgrading to Graylog 1.0 GA</title>
      <link>https://secopsmonkey.com/post/2015-02-26-upgrading_to_graylog_1.0_ga/</link>
      <pubDate>Thu, 26 Feb 2015 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2015-02-26-upgrading_to_graylog_1.0_ga/</guid>
      
        <description>&lt;p&gt;A few things to keep in mind. First a name change. During the 0.X versions the software was called
Graylog2. Starting with the release of 1.0 the name was changed go Graylog. I&amp;rsquo;m pretty happy about
the change. Graylog2 was a bit of a mouthful and including version numbers in package names has
always annoyed me anyway. However, changing the version number makes the upgrade a bit harder.&lt;/p&gt;
&lt;p&gt;I would plan this upgrade for a time when you can expect to not receive logs. During part of the
upgrade you will have two instances of Graylog installed which may conflict with each other.&lt;/p&gt;
&lt;h1 id=&#34;package-repository&#34;&gt;Package Repository&lt;/h1&gt;
&lt;p&gt;If you followed my previous posts you know I deal with Ubuntu, but modifying the steps to something
more RedHaty should be relatively easy. First thing&amp;rsquo;s first, we have to change where we download
the packages. Edit &lt;code&gt;/etc/apt/sources.list.d/graylog2.list&lt;/code&gt; to look more like this&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-cfg&#34; data-lang=&#34;cfg&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;#file generated by puppet&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# This is the TORCH hosted official repositories for Graylog2&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;deb https://packages.graylog2.org/repo/debian/ trusty 1.0&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;With this version they changed to HTTPS for package delivery so make sure you also have the
&lt;code&gt;apt-transport-https&lt;/code&gt; package installed.&lt;/p&gt;
&lt;h1 id=&#34;install-the-new-packages&#34;&gt;Install the New Packages&lt;/h1&gt;
&lt;p&gt;Since the name of the software changed the packages also changed. This one&amp;rsquo;s easy. Just do a&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;aptitude install graylog-web graylog-server&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;At this point we have both Graylog2 0.{whatever} and Graylog 1.0 installed.&lt;/p&gt;
&lt;h1 id=&#34;update-graylog-server-config&#34;&gt;Update Graylog Server Config&lt;/h1&gt;
&lt;p&gt;The configuration file path is very similar between the two, which makes it easy. Historically
&lt;code&gt;graylog2-server&lt;/code&gt; installed its config files in one of two locations. The primary config file
is &lt;code&gt;/etc/graylog2.conf&lt;/code&gt; and all other files are in &lt;code&gt;/etc/graylog2/server/&lt;/code&gt;. Starting with
Graylog 1.0 all server config files are stored in &lt;code&gt;/etc/graylog/server/&lt;/code&gt;. The quick and dirty
approach is to just&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;cp /etc/graylog2.conf /etc/graylog/server/server.conf&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;I would actually recommend checking a diff of the files first and make sure you don&amp;rsquo;t need to
update anything manually.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;diff -y /etc/graylog2.conf /etc/graylog/server/server.conf
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Look for changes that need to be made&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;cp /etc/graylog2.conf /etc/graylog/server/server.conf
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Clean up new server.conf with any necessary modifications&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;There were a few references to &lt;code&gt;graylog2&lt;/code&gt; in file paths that needed to be corrected, but there was
only one setting that desperately needed changed. With Graylog 1.0 there&amp;rsquo;s a new option called
a message journal that is used. If you don&amp;rsquo;t specify a location in the config file it will default
to &lt;code&gt;/data/journal/&lt;/code&gt; which may result in startup errors. I plopped it in the same corresponding
location as the old message cache.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-cfg&#34; data-lang=&#34;cfg&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Enable the disk based message journal.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;message_journal_enabled&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;true&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# The directory which will be used to store the message journal&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# must not contain any other files than the ones created by Graylog&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;message_journal_dir&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;/var/lib/graylog-server/journal&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Also make sure to copy over the &lt;code&gt;node-id&lt;/code&gt;. Otherwise your Graylog 1.0 instance will show up as a
new server node which could cause some problems down the road.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;cp /etc/graylog2/server/node-id /etc/graylog/server/node-id&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;h1 id=&#34;update-graylog-web-config&#34;&gt;Update Graylog Web Config&lt;/h1&gt;
&lt;p&gt;The default web server configuration file name has changed from &lt;code&gt;graylog2-web-interface.conf&lt;/code&gt; to
&lt;code&gt;web.conf&lt;/code&gt; which better matches Graylog Server. We&amp;rsquo;re going to follow much the same process as
above. First validate the web server configs and copy them over.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;diff -y /etc/graylog2/web/graylog2-web-interface.conf /etc/graylog/web/web.conf
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Look for changes that need to be made&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;cp /etc/graylog2/web/graylog2-web-interface.conf /etc/graylog/web/web.conf
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Clean up new web.conf with any necessary modifications&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;On my system no other files needed modification, but you might want to check them out as well.&lt;/p&gt;
&lt;h1 id=&#34;recycle-services&#34;&gt;Recycle Services&lt;/h1&gt;
&lt;p&gt;Along with everything else the service names also changed. Cycle all the services to get the new
instance up and running.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;initctl stop graylog2-server
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;initctl stop graylog2-web
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;initctl start graylog-server
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;initctl start graylog-web&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;h1 id=&#34;cleanup&#34;&gt;Cleanup&lt;/h1&gt;
&lt;p&gt;Since the service account user changed we&amp;rsquo;ll go ahead and change ownership to the new user.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;find / -user graylog2 -print0 &lt;span class=&#34;p&#34;&gt;|&lt;/span&gt; xargs -0 chown graylog:graylog
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;find / -user graylog2-web -print0 &lt;span class=&#34;p&#34;&gt;|&lt;/span&gt; xargs -0 chown graylog-web:graylog-web&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Next kill all the old packages.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;aptitude remove graylog2-server graylog2-web&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Now delete all the old config files. Make sure you have backups, just in case.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;rm -rf /etc/graylog2 /etc/graylog2.conf /etc/init/graylog2-server.conf /etc/init/graylog2-web.conf /etc/default/graylog2-server /etc/default/graylog2-web&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Now we can kill all the old caches.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;rm -rf /var/lib/graylog2-server/ /var/lib/graylog2-web/&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Once your local log retention schedule has expired to ahead and wipe the application logs as well.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;rm -rf /var/log/graylog2-web /var/log/graylog2-server&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Now kill the old service accounts. They&amp;rsquo;re completely unecessary at this point and can be removed.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;userdel graylog2
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;userdel graylog2-web&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;h1 id=&#34;finalize&#34;&gt;Finalize&lt;/h1&gt;
&lt;p&gt;At this point we should be totally good to go with our new environment. Log in to the web interface
and browse around. Make sure the inputs are listening where they should be. Verify that data is
actually accessible. If not, well, I wish you the best of luck in your exciting new troubleshooting
endeavors!&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>A Retention Complication</title>
      <link>https://secopsmonkey.com/post/2015-02-13-a-retention-complication/</link>
      <pubDate>Fri, 13 Feb 2015 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2015-02-13-a-retention-complication/</guid>
      
        <description>&lt;p&gt;After going through the &lt;a href=&#34;http://secopsmonkey.com/my-thoughts-on-retention.html&#34;&gt;retention schedule exercise&lt;/a&gt; on our infrastructure log management
system I ran into a bit of an interesting situation. First, some background.&lt;/p&gt;
&lt;p&gt;Historically Graylog2 hasn&amp;rsquo;t provided any capability to perform time based retention, which is
pretty dang lame. Instead all retention was performed based purely on message count. If you have
good figures on your log generation rates this is no big deal. Let&amp;rsquo;s assume you have a 6 month
retention schedule and your environment has leveled out to about 25,000 events per second. This
gives us 36,000,000 events per day at 182, which in Graylog2 is pretty easy to set up. Just change
these lines in your &lt;code&gt;/etc/graylog2.conf&lt;/code&gt; and restart the service.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;elasticsearch_max_docs_per_index&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;mi&#34;&gt;36000000&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;elasticsearch_max_number_of_indices&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;mi&#34;&gt;182&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;But, in reality, there&amp;rsquo;s a kind of a hitch. Most of us don&amp;rsquo;t live in that predictable of an
environment. Most of us are actively building out our logging infrastructure and frequently
adding new systems that have never been profiled with no clear understood volume. Or we&amp;rsquo;re a
dynamic shop with systems coming up and down automagically to meet demand. We might go from
200 EPS one day to 700 the next. Growth can be approximated over time but it takes significant
data to be meaningful.&lt;/p&gt;
&lt;p&gt;Fortunately for us Graylog2 &lt;em&gt;finally&lt;/em&gt; started &lt;a href=&#34;https://www.graylog.org/graylog2-v0-92/&#34;&gt;shipping time based retention&lt;/a&gt;. I was pretty
stoked and jumped right in, especially since it was as easy as two config options.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;elasticsearch_max_time_per_index&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;mi&#34;&gt;1&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;d&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;elasticsearch_max_number_of_indices&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;mi&#34;&gt;20&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Personally I didn&amp;rsquo;t care how many indices there were just so long as the oldest one was no more
6 months old. So I set &lt;code&gt;elasticsearch_max_time_per_index&lt;/code&gt; to 182, thinking any index older than
that would get deleted, and outright ignored &lt;code&gt;elasticsearch_max_number_of_indices&lt;/code&gt; entirely. Therein
lies my fatal flaw. As it turns out I was a bit off. The option &lt;code&gt;elasticsearch_max_time_per_index&lt;/code&gt; is used
for rotation whereas &lt;code&gt;elasticsearch_max_number_of_indices&lt;/code&gt; is used for retention. This is important.&lt;/p&gt;
&lt;h1 id=&#34;retention-is-not-rotation&#34;&gt;Retention Is Not Rotation&lt;/h1&gt;
&lt;p&gt;I mean, this does make sense. Retention is how long we keep something around and rotation is how
often we fiddle with it. After a good bit of &lt;a href=&#34;https://github.com/Graylog2/graylog2-server/issues/871&#34;&gt;back and forth&lt;/a&gt; with the developers it occurred to
me that I&amp;rsquo;ve mentally combined the two actions into a single concept. Take for example log files on
a Linux system. A very common setup is to rotate files nightly and delete them weekly. I realized
that I&amp;rsquo;ve internalized the rotation strictly as a resource management problem. By having files
that contain logs from a single day it&amp;rsquo;s easier to read, they&amp;rsquo;re generally smaller, and all around
easier to work with.&lt;/p&gt;
&lt;p&gt;Retention, again in my head, however is much more important. Here we&amp;rsquo;re not really concerned about
ease of use but rather importance. We keep files because they&amp;rsquo;re important. Then, after a certain
time, they&amp;rsquo;re not or we&amp;rsquo;re told we can&amp;rsquo;t. Either way, we delete them because they&amp;rsquo;re not important
anymore&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a href=&#34;#fn:1&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;By merging the two concepts I had relegated rotation to be just something we do as &lt;em&gt;part&lt;/em&gt; of
retention because it&amp;rsquo;s easy, not because it has any actual merit. This is where I had failed, and
what I think bears repeating. Rotation is important and has its own merits. Rotation is how we
organize log files. Rotation is how we heat tier information. Hell, rotation is how I organize the
notebook on my desk.&lt;/p&gt;
&lt;p&gt;Normally I find myself rethinking technical implementations as a result of policy/procedure/governance
enlightenment. This incident is the opposite. Thanks to a moment of technical implementation
enlightenment I&amp;rsquo;ve found myself re-thinking some policies and procedures and how there might be
missing pieces.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;Ok, fine, often times we delete things not because they&amp;rsquo;re not important but because we don&amp;rsquo;t have
the storage. Don&amp;rsquo;t fool yourself. If it was &lt;em&gt;really&lt;/em&gt; that important we would find the storage. If something
truly is important, but an extra drive tray for the NAS is denied, we would find a way.&amp;#160;&lt;a href=&#34;#fnref:1&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>Better Use of Office 365 as a Smart Host with Postfix</title>
      <link>https://secopsmonkey.com/post/2015-01-16-better-mail-relaying-postfix-through-office-365/</link>
      <pubDate>Fri, 16 Jan 2015 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2015-01-16-better-mail-relaying-postfix-through-office-365/</guid>
      
        <description>&lt;p&gt;A while ago I wrote a post on how I managed to get my systems integrated with &lt;a href=&#34;http://secopsmonkey.com/mail-relaying-postfix-through-office-365.html&#34;&gt;Office365 to send
email notifications&lt;/a&gt;. At the time the method I used worked well enough but it was annoying.
Every system had to log in individually which meant the username and password had to be distributed
to every system. Certainly not my favorite thing.&lt;/p&gt;
&lt;p&gt;Since then I&amp;rsquo;ve learned that Office 365 has something called &amp;ldquo;Connectors&amp;rdquo;. A Connector is something
like a VPN tunnel in that it allows you to send mail through a trusted connection&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a href=&#34;#fn:1&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;. The idea
being that if you can connect to the connector then you&amp;rsquo;ve already been sufficiently proofed to do
fun things.&lt;/p&gt;
&lt;p&gt;What really attracted me to the idea of one of these Connectors was a slight change in my use case.
We brought an application on board that sent emails directly instead of relying on the system
mailer. This wasn&amp;rsquo;t overly surprising, but the real sticking point was that it also attempted to
perform a Send-As. One of the caveats from the previous post was that all emails had to be sent
from the user account that was being used for authentication. All of a sudden our previously
working system wasn&amp;rsquo;t. However, by using a connector the mailer is considered fully trusted and
can send emails from any address, so long as it matches the relevant domain.&lt;/p&gt;
&lt;p&gt;In order to account for the new app I opted to redesign my services mail flow a bit. Now instead
of every host authenticating directly to Office 365 I will have a single server whose dedicated
function is to act as a mail relay. All services will use the mail relay as their smarthost and
the mail relay will connect to Office 365 using a connector.&lt;/p&gt;
&lt;h1 id=&#34;create-the-connector&#34;&gt;Create the Connector&lt;/h1&gt;
&lt;ol&gt;
&lt;li&gt;Log into your Office 365 instance using an Administrator account.&lt;/li&gt;
&lt;li&gt;From the main Dashboard expand &amp;lsquo;Mail&amp;rsquo; on the left hand side and click on &amp;lsquo;Exchange&amp;rsquo;.
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2015-01-16-better-mail-relaying-postfix-through-office-365/O365_RELAY_Step_1_Dashboard.jpg&#34;
    alt=&#34;Open Exchange&#34;&gt;
&lt;/figure&gt;
&lt;/li&gt;
&lt;li&gt;This will bring us to the main Exchange management screen, from here we want to click on &amp;lsquo;Connectors&amp;rsquo;.
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2015-01-16-better-mail-relaying-postfix-through-office-365/O365_RELAY_Step_2_Exchange_Admin.jpg&#34;
    alt=&#34;Open Connectors&#34;&gt;
&lt;/figure&gt;
&lt;/li&gt;
&lt;li&gt;Click the &amp;lsquo;+&amp;rsquo; to add a new connector.
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2015-01-16-better-mail-relaying-postfix-through-office-365/O365_RELAY_Step_3_Connectors.jpg&#34;
    alt=&#34;New Connectors&#34;&gt;
&lt;/figure&gt;
&lt;/li&gt;
&lt;li&gt;Give it a fluffy name, this is purely an identifier so you can recognize the connector later.&lt;/li&gt;
&lt;li&gt;Select &amp;lsquo;On-premises&amp;rsquo; since our relay is on our site.&lt;/li&gt;
&lt;li&gt;Under &amp;lsquo;Connection Security&amp;rsquo; make sure &amp;lsquo;Force TLS&amp;rsquo; is selected.&lt;/li&gt;
&lt;li&gt;Under &amp;lsquo;Domain Restrictions&amp;rsquo; we have a couple of options. &amp;ldquo;None&amp;rdquo; means anyone can use this connector, &amp;ldquo;Restrict domains by certificate&amp;rdquo; means we&amp;rsquo;ll use an x509 certificate as authentication, and &amp;ldquo;Restrict domains by IP address&amp;rdquo; means we&amp;rsquo;ll only allow specific IP addresses to use the connector. For my purposes I want to use certificate so we&amp;rsquo;ll select &amp;lsquo;Restrict domains by certificate&amp;rsquo;. Under
&amp;lsquo;Certificate&amp;rsquo; enter the CN that will be used by the local mail relayer.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Appropriately choosing your &amp;lsquo;Domain Restrictions&amp;rsquo; is pretty important. Restricting by IP address can
be pretty not bad, but if you&amp;rsquo;re in a NATed environment be careful you don&amp;rsquo;t accidentally increase
your scope. In my case I didn&amp;rsquo;t want to put my mail relayer in our DMZ, so it was 1-to-many NATed,
and I actually have a certificate I can use for this host. Those two considerations made choosing
&amp;ldquo;by certificate&amp;rdquo; pretty much a no brainer.&lt;/p&gt;
&lt;p&gt;The last option is &amp;ldquo;Associated accepted domains&amp;rdquo;. If you try to add anything here you&amp;rsquo;ll be shown
a list of all domains you have management rights in Office 365. If you add any domains here then
the connector will reject any emails that are destined for another domain. If your services will
&lt;em&gt;only&lt;/em&gt; be mailing internally then this setting may work for you. If you do anything externaly,
like email based alerting with a hosted provider, you&amp;rsquo;ll want to leave it empty.&lt;/p&gt;
&lt;h1 id=&#34;configure-the-mail-relay-host&#34;&gt;Configure the Mail Relay Host&lt;/h1&gt;
&lt;p&gt;This assumes you already have a server you want to use as your mail relay system. I wanted one
so I made one. Building such a host is beyond the scope of this post and is left as an exercise
for the reader.&lt;/p&gt;
&lt;p&gt;On your mail relay host edit &lt;code&gt;/etc/postfix/main.cf&lt;/code&gt; so that it looks like:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-cfg&#34; data-lang=&#34;cfg&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;relayhost&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;[domain-com.mail.protection.outlook.com]&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;smtp_sasl_security_options&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;noanonymous&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;smtp_sasl_tls_security_options&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;noanonymous&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;smtp_tls_security_level&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;encrypt&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;smtp_use_tls&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;yes&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;smtpd_tls_cert_file&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;/path/to/private/cert&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;smtpd_tls_key_file&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;/path/to/private/key&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;mynetworks&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;127.0.0.0/8, 10.0.0.0/8&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;The &lt;code&gt;relayhost&lt;/code&gt; name is a bit odd. Office 365 always creates the DNS name for you but leaves it in
the format &lt;code&gt;${fqdn}.mail.protection.outlook.com&lt;/code&gt; where any periods of your email domain in the
&lt;code&gt;fqdn&lt;/code&gt; are replaced with hyphens. That is, if &lt;code&gt;secopsmonkey.com&lt;/code&gt; were hosted by Office 365 I would
use &lt;code&gt;secopsmonkey-com.mail.protection.outlook.com&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;We&amp;rsquo;re also using the &lt;code&gt;mynetworks&lt;/code&gt; option for access restriction. This is how we tell postfix which
IP addresses are allowed to send email to it. In the above example we&amp;rsquo;re telling the relay host
that any host within 127.0.0.0/8 or 10.0.0.0/8 are allowed to send email to it. You should change
those values to be your trusted environment.&lt;/p&gt;
&lt;p&gt;Once you restart postfix it will be able to connect through the connector using the x509 certificate
for authentication. Now any messages sent from this server will be routed through Office 365 by way
of the connector.&lt;/p&gt;
&lt;h1 id=&#34;configure-the-endpoints&#34;&gt;Configure the Endpoints&lt;/h1&gt;
&lt;p&gt;As for the client machines we just need to point them at the mail relay host.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-cfg&#34; data-lang=&#34;cfg&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;relayhost&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;[mailrelay.domain.com]:25&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;smtp_sasl_auth_enable&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;no&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;smtp_tls_security_level&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;may&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;na&#34;&gt;smtp_sasl_security_options&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;noanonymous&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Once we restart &lt;code&gt;postfix&lt;/code&gt; it will start relaying through the mail relayer which authenticates to
Office 365 through the connector.&lt;/p&gt;
&lt;p&gt;Once all this is set up we&amp;rsquo;re freed from the earlier restriction of having to rewrite the from
addresses of all of our emails. It also means we have a single choke point to monitor and manage
for our outgoing messages.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;Ok it&amp;rsquo;s &lt;em&gt;really&lt;/em&gt; not so much a secured tunnel as it is an assumption of authenticity. In that way it acts more like the REMOTE_USER field in a web server.&amp;#160;&lt;a href=&#34;#fnref:1&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>Running SSH on Non-Default Ports</title>
      <link>https://secopsmonkey.com/post/2015-01-09-non-default-ssh-ports/</link>
      <pubDate>Fri, 09 Jan 2015 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2015-01-09-non-default-ssh-ports/</guid>
      
        <description>&lt;p&gt;Earlier this week I saw this tweet.&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2015-01-09-non-default-ssh-ports/jorgensen_twitter_post.png&#34;
    alt=&#34;Non-Standard SSH Ports?&#34;&gt;
&lt;/figure&gt;

&lt;p&gt;There are a few topics that seem to come up with some regularity and running applications on
non-standard points is definitely one of them. Like everyone else I have some opinions.&lt;/p&gt;
&lt;p&gt;The Internet is a wild and scary place, full of malcontents whose motives range from curiosity all
the way to criminal enterprise. These unsavories are constantly scanning for computers running
services they hope to exploit; usually the more common services such as SSH, HTTP, FTP, etc. The
scans typically fall into one of two categories:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Recon scans to see what IP address have those services open.&lt;/li&gt;
&lt;li&gt;Exploit scans against IP addresses who have been found to be running a specific service.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Considering how large the Internet is it is typically unfeasible to look on every port of every IP
address to find what&amp;rsquo;s listening everywhere. This is the crux of the advice to change your default
port. If these disaffected individuals want to find SSH servers they will start probing each IP
address on port 22 (they may also add some common alternates such as 222 or 2222). Then, once they
have their list of IP addresses with port 22 open, they will start their password brute force to
guess usernames/passwords or launch their exploit kit of choice and start testing known (at least
to them) vulnerabilities on the target system.&lt;/p&gt;
&lt;p&gt;This means that if you change your SSH port to 34887 then that sweep will pass you on by, likely
resulting in you not being targeted by the followup break-in.&lt;/p&gt;
&lt;p&gt;Seems rosy right? There are some disadvantages though.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Client Support: Everybody who connects to your server will need to know and use the changed port. If you are in a heavily managed environment, this configuration can be pushed down to the clients, or if you have few enough users it should be easy to communicate.&lt;/li&gt;
&lt;li&gt;Documentation Exceptions: Most network devices, such as firewalls and IDSes, are pre-setup for common services to be run on common ports. Any firewall rules related to this service on this device will need to be inspected and possibly modified. Similarly, IDS signatures will be tweaked so as to only perform SSH inspection on port 22. You will need to modify every signature, every time they are updated, with your new port. (As a data point there are currently 136 VRT and ET snort signatures involving SSH).&lt;/li&gt;
&lt;li&gt;System Protections: Modern Linuxes often ship with an kernel layer MAC and/or RBAC systems (e.g. SELinux on RedHat based or AppAmor on Debian based) and that are designed to only allow applications to do exactly what they&amp;rsquo;re intended to do. That could range from accessing the &lt;code&gt;/etc/hosts&lt;/code&gt; file, to writing to a specific file, or sending a packet out on the network. Depending on how this system is configured it may, by default, forbid &lt;code&gt;sshd&lt;/code&gt; from binding to a non-standard port. You would need to maintain a local policy that would allow it.&lt;/li&gt;
&lt;li&gt;Other Party Monitoring: If you have an external Information Security division, or outsource monitoring, then they will need to be made aware of the change. When performing a security assessment, or analyzing logs looking for security threats, if I see an SSH server running on a non-standard port (or an SSH server on a non-UNIX/Linux for that matter) I treat it as a potential backdoor and invoke the compromised system part of incident handling procedure. Sometimes it is resolved in 5 minutes after making a call to the administrator and being told it&amp;rsquo;s legitimate, at which point I update documentation, other times it really is badness that gets taken care of. In any event, this can result in down-time for you or, at the least, a nerve racking call when you answer your phone and hear, &amp;ldquo;Hi, this is Bob from the Information Security Office. I have a few questions for you.&amp;rdquo;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Before changing your port you need to take all of this into account so you know you&amp;rsquo;re making the
best decision. Some of those disadvantages may not apply, but some certainly will. Also consider
what you&amp;rsquo;re trying to protect yourself against. Often times it is simply easier to just configure
your firewall to only allow access to 22 from specific hosts, as opposed to the whole Internet.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>My Thoughts on Retention</title>
      <link>https://secopsmonkey.com/post/2015-01-02-my-thoughts-on-retention/</link>
      <pubDate>Fri, 02 Jan 2015 00:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2015-01-02-my-thoughts-on-retention/</guid>
      
        <description>&lt;p&gt;Sometimes I feel like Records Retention is the red-headed step child. It&amp;rsquo;s obviously important,
almost every regulation that covers us talks about it to one extent or another. There&amp;rsquo;s a base
assumption under the major frameworks that retention is happening&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a href=&#34;#fn:1&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;. Having a policy around
records management is even a requirement under SOX, and for certain classifications of data under
PCI-DSS and HIPAA. Despite all this, based on my experience, retention schedules are often amongst
the last policies to be built and the least likely to be enforced.&lt;/p&gt;
&lt;p&gt;Personally, I find this somewhat backwards. Instituting a Records Retention policy is one of the
easier policies to write. At its core a retention schedule is not much more than a list of data
types and when they&amp;rsquo;re to be disposed of. The trick is figuring out what those time frames should
be. For each data type you need to take into account at least:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Liability&lt;/strong&gt; — We cannot be held liable for any data that falls outside of our published retention schedules (unless the published schedule contradicts regulation or legislation).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Storage&lt;/strong&gt; — Keeping stuff a long time means we have to pay to keep it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Processing&lt;/strong&gt; — The longer back we keep data, particularly logs and alerts, the longer it generally takes to run reports and process the data.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Usability&lt;/strong&gt; — We need to understand how we’re going to use the information so we can know out how long we actually need to keep it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Compliance&lt;/strong&gt; — Most regulations, laws, and standards define minimum storage times, sometimes you’ll see maximums. Whatever else we need to make sure we follow these, or else get someone higher than us to sign off on ignoring the requirement.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Generally speaking liability, storage, and processing lean towards keeping data as short a time
as possible. Usability and compliance can set specific maximums or minimums.&lt;/p&gt;
&lt;p&gt;This is when you need to actually start talking to people. From your compliance team you need to
find out what kind of data needs to be collected and what time frames may be prescribed based on
how your product or company is regulated. From the data custodians you need to know the technical
limitations of how much can be stored and how. From the users&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a href=&#34;#fn:2&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt; you need to find out what kinds
of information is being used and &lt;em&gt;how&lt;/em&gt; it&amp;rsquo;s being used. For example,&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Syslogs are primarily useful for system state monitoring, troubleshooting, and auditing. System
state is very transient information that is rarely useful beyond a day or so. Problem generally
make themselves apparent quickly and can be resolved with relatively short lived history. It is
very rare to go back more than 3-4 weeks during troubleshooting. Audit reports are run regularly
against the logs regularly and can be used without referring to the authoritative source.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;This tells us we minimally need 4 weeks but can probably scale out to 3-6 months in order to cover
abnormal situations. One year or longer is almost definitely extreme overkill.&lt;/p&gt;
&lt;p&gt;Or to beat that horse using a business side example,&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Client engagements are charged on a per encounter basis and billed monthly. Notes are kept for
each engagement and are used to clarify potential discrepancies in the event of a billing
dispute. The notes do not contain any legislated or contractually defined data but does contain
private information regarding the client. Billing disputes are rare and frequently initiated
within 2 months and have never been resolved later than 9 months following the engagement.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;In this situation there would be a business need to keep the notes at least  9 months, we can
round that up to 12 to make a nice easy number. However, the actual billing data may need to be
kept longer depending on any number of factors such as financial reporting or tax filings.&lt;/p&gt;
&lt;p&gt;There are a lot of variables involved and there may be some political negotiation, if billing needs
12 months and storage has only allocated disk for 5 then someone will have to change something.
That&amp;rsquo;s not to say you have to get it all right at the outset either. Create a default schedule
and add to it as you define the data elements. Just get something in place. Your technical team
will be happy because their storage stops filling up and your legal team will be happy when they
receive a subpoena for &amp;ldquo;all network traffic, emails, and telephone records related to John Snow
since August 10th, 1995&amp;rdquo;.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;Under NIST SP800-53r4 check out AU-11, SI-12, CM-2(3). Or if you prefer something drier than NIST go to ISO 15489-1:2001.&amp;#160;&lt;a href=&#34;#fnref:1&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;Users is a pretty generic term here. For things like firewall, web, or auth logs it might be your systems administrators or security analysts. For things like client engagements and front desk visitor check-in it may be your billing or business intelligence folk.&amp;#160;&lt;a href=&#34;#fnref:2&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>Seccubus on Ubuntu - The Missing Manual</title>
      <link>https://secopsmonkey.com/post/2014-11-21-seccubus-on-ubuntu-the-missing-manual/</link>
      <pubDate>Fri, 21 Nov 2014 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-11-21-seccubus-on-ubuntu-the-missing-manual/</guid>
      
        <description>&lt;p&gt;My tool of choice for vulnerability scanning has always been Nessus, going all the way back to when
it was properly OpenSource and the &amp;lsquo;Experimental Checks&amp;rsquo; checkbox was a sure fire way to crash your
target. Since going full commercial it&amp;rsquo;s only gotten better and the current interface is very clean
and polished. The biggest downside being that each scan is treated as a stand-alone unit so the
application itself can&amp;rsquo;t do any kind of trending or inter-scan analysis. The company does provide
a tool that does exactly this but monies. Compared to many tools it&amp;rsquo;s still somewhat cheap, but I
can still think of many places I would rather spend the funds.&lt;/p&gt;
&lt;p&gt;In past work we did a bit of custom development to perform the tasks; they&amp;rsquo;re kind enough to
provide a REST API to the scanners, and XML reports means we can do XML-&amp;gt;CSV for aggregate
reporting. Every now and then we would look for more of a platform tool to do the task. For some
time I hoped InProtect would be that tool but it died on the vine and never quite made it to
useful.&lt;/p&gt;
&lt;p&gt;Lately I&amp;rsquo;ve been looking at a tool called &lt;a href=&#34;https://www.seccubus.com&#34;&gt;Seccubus&lt;/a&gt;. Much like InProtect tries to fill that
management and reporting platform niche without costing a good sized fraction of an FTE. In
theory this will give me single web interface to schedule the scans of hosts and track how the
state of the host changed over time. Or, even better, let me quickly carve up a whole big whack
of systems looking for specific things. Since I&amp;rsquo;m a team of one I could, in theory, take care
of this myself using the same methods we&amp;rsquo;ve done in the past and not worry about fancy tools.
Excel is still pretty nifty for sorting and aggregating data. Seccubus really drew my eye for two
reasons.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;It supports more than just Nessus (Nikto, Skipfish, and Nmap)&lt;/li&gt;
&lt;li&gt;It&amp;rsquo;s actually under active development&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Even if the interface never gets as slick as the commercial tools knowing a group of people
are actually making code commits tells me I don&amp;rsquo;t have to worry as much about picking up a tool
and having to internally maintain it when they get disappear.&lt;/p&gt;
&lt;p&gt;Unsurprisingly the documentation is pretty rough. Most of it is dated 2011 and lacking in some
pretty substantial areas. The install docs date from a time when version 2 was still in beta
and don&amp;rsquo;t address &lt;code&gt;.deb&lt;/code&gt; packages at all. In a perfect world the packages are complete enough
that installation instructions are &amp;ldquo;Run &lt;code&gt;apt-get install seccubus&lt;/code&gt; and go to the webpage.&amp;rdquo;.
Unfortunately I found it to be not quite so simple.&lt;/p&gt;
&lt;h1 id=&#34;installation&#34;&gt;Installation&lt;/h1&gt;
&lt;p&gt;My test system is running Ubuntu 14.04, which works well enough since they give me a nice
&lt;code&gt;.deb&lt;/code&gt; to install from. What they &lt;em&gt;don&amp;rsquo;t&lt;/em&gt; do is provide any documentation as to the pre-reqs.
The deb file does contain &lt;em&gt;most&lt;/em&gt; of the necessary pre-reqs but not all.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;aptitude install apache2 mysql-server ruby libalgorithm-diff-perl libjson-perl libxml-simple-perl libhtml-tree-perl libapache2-mod-perl2
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;wget https://github.com/schubergphilis/Seccubus_v2/releases/download/2.10/seccubus_2.10.B194_all.deb
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;dpkg -i seccubus_2.10.B194_all.deb&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;h1 id=&#34;database-configuration&#34;&gt;Database Configuration&lt;/h1&gt;
&lt;p&gt;The deb package will copy out all the files but it won&amp;rsquo;t do any of the database set up for us.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;mysql -u root -p &lt;span class=&#34;s&#34;&gt;&amp;lt;&amp;lt; EOF
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;s&#34;&gt;create database seccubus;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;s&#34;&gt;grant all privileges on seccubus.* to seccubus@localhost identified by &amp;#39;seccubus&amp;#39;;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;s&#34;&gt;flush privileges;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;s&#34;&gt;EOF&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;mysql -u seccubus -pseccubus seccubus &amp;lt; /var/lib/seccubus/structure_v5.mysql
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;mysql -u seccubus -pseccubus seccubus &amp;lt; /var/lib/seccubus/data_v5.mysql&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;It may be somewhat smart to set a different password for the MySQL seccubus user, but when you
do make sure to change it in the &lt;code&gt;/etc/seccubus/config.xml&lt;/code&gt; file as well.&lt;/p&gt;
&lt;h1 id=&#34;web-server-setup&#34;&gt;Web Server Setup&lt;/h1&gt;
&lt;p&gt;The package drops an attempt at a virtual host config in &lt;code&gt;/etc/apache2/conf.d/seccubus.conf&lt;/code&gt;.
This doesn&amp;rsquo;t get loaded automatically, and it also places it in a janky URI. I&amp;rsquo;m using this
host as a dedicated instance so the only web app being served here is Seccubus. To make it
easy on myself I edited &lt;code&gt;/etc/apache2/sites-available/000-default.conf&lt;/code&gt; to be:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-apache&#34; data-lang=&#34;apache&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;&amp;lt;VirtualHost&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;*:80&lt;/span&gt;&lt;span class=&#34;nt&#34;&gt;&amp;gt;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;nb&#34;&gt;ServerAdmin&lt;/span&gt; webmaster@localhost
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;nb&#34;&gt;DocumentRoot&lt;/span&gt; &lt;span class=&#34;sx&#34;&gt;/opt/seccubus/www/&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;nb&#34;&gt;LogLevel&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;info&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;nb&#34;&gt;ErrorLog&lt;/span&gt; ${APACHE_LOG_DIR}/error.log
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;nb&#34;&gt;CustomLog&lt;/span&gt; ${APACHE_LOG_DIR}/access.log combined
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;nt&#34;&gt;&amp;lt;Location&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;nt&#34;&gt;&amp;gt;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;nb&#34;&gt;Allow&lt;/span&gt; from &lt;span class=&#34;k&#34;&gt;all&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;nb&#34;&gt;Require&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;all&lt;/span&gt; granted
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;c&#34;&gt;# Allow from .example.com&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;nb&#34;&gt;AddHandler&lt;/span&gt; cgi-script .pl
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;nb&#34;&gt;Options&lt;/span&gt; ExecCGI
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;nt&#34;&gt;&amp;lt;/Location&amp;gt;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;&amp;lt;/VirtualHost&amp;gt;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;# vim: syntax=apache ts=4 sw=4 sts=4 sr noet&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Additionally, we need to enable a couple of modules or else the web application will not throw
any visible errors but will only be partially functional.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;a2enmod perl
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;a2enmod cgi&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;h1 id=&#34;points-to-note&#34;&gt;Points to Note&lt;/h1&gt;
&lt;p&gt;The Seccubus application has no internal authentication or user management. I think it goes without
saying that given the sensitivity of the information you&amp;rsquo;ll be storing you should get some. Since
Apache has many easy to use authentication backends you should look there. Which also means setting
up HTTPS so you&amp;rsquo;re passing credentials, or scan results for that matter, in clear-text.&lt;/p&gt;
&lt;p&gt;Since there is no user management we&amp;rsquo;re also stuck with access == kingdom keys. Unlike the
commercial tools or InProtect^(RIP) it&amp;rsquo;s impossible to restrict access to allowed subnets or
designate reporting users.&lt;/p&gt;
&lt;p&gt;All in all the application has some nice starting points but feels very much like an early work in
progress. I think it&amp;rsquo;ll work fine for my uses, but I&amp;rsquo;m a lone security guy with a couple of
sysadmins. I feel as if there&amp;rsquo;s quite a lot of work to be done before I&amp;rsquo;d suggest it for enterprise
use. In my opinion the minimum features to make it out of test lab, or my scale, are task granularity
for user permissions and assigning network scopes to specific scanners. I want to be able to grant
read only access to report viewing, specifically only for those hosts you&amp;rsquo;re responsible for. I
want to be able to allow admins to initiate scans of their own equipment using settings I&amp;rsquo;ve
vetted. I want to be able to place a scanner in a remote site and assign that site&amp;rsquo;s network block
to the remote scanner so we don&amp;rsquo;t flood the up-link with scans.&lt;/p&gt;
&lt;p&gt;The largest stumbling block I ran into was that the application didn&amp;rsquo;t seem to function correctly
in my installation of Chrome (running 38.0.2125.111 on Mac OSX 10.10). I haven&amp;rsquo;t gotten far enough
along to know if this is an application bug or a browser issue. I&amp;rsquo;m including the note as a heads
up rather than a warning.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>Migrating Graylog Servers - Part 6 - Lessons Learned</title>
      <link>https://secopsmonkey.com/post/2014-11-14-migrating-graylog2-servers-part-6-lessons-learned/</link>
      <pubDate>Fri, 14 Nov 2014 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-11-14-migrating-graylog2-servers-part-6-lessons-learned/</guid>
      
        <description>&lt;p&gt;&lt;em&gt;This is the sixth and final post in a multi-part series where I explore the process of transforming
an existing Graylog install into a resilient and scalable multi-site installation.
&lt;a href=&#34;http://secopsmonkey.com/migrating-graylog2-servers.html&#34;&gt;Start here for Part 1&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Let me start by saying that the entire process was a huge learning experience for me. Over the
course of my career I have dealt with a number of different logging systems of various degrees
of complexity. The simplest being straight syslog, syslog-ng or rsyslog, logging to text files
with analysis in awk/sed/wc/grep/perl. The most complex being commercial solutions like RSA
enVision and NitroView, now McAfee ESM. The simple solutions tend to be lacking in more advanced
reporting features but are insanely simple to manage. The more complex will let you report or
alert on almost condition you can consider but at the expense of requiring intense amounts of
time to configure and manage.&lt;/p&gt;
&lt;p&gt;All that being said, log management was never a primary focus of mine. In past jobs those duties
did fall within my team and I was always closely involved in the process. As a result I do have
a strong appreciation of many of the complexities involved in developing a log management/analysis
program and some knowledge of the players in the field. I&amp;rsquo;ve been aware of Graylog2 for some time
but this was certainly my first opportunity to really dig into it and also my first time working
with most all of the technologies it relies on.&lt;/p&gt;
&lt;p&gt;I came into the entire process as close to a blank slate as possible. I had no opinion of the
product. I&amp;rsquo;ve never worked with any of the developers&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a href=&#34;#fn:1&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;. I&amp;rsquo;ve never used any of the underlying
technologies involved. I came to this purely as an engineer who wanted a system that didn&amp;rsquo;t suck
and would let him do incident response and auditing. This is why I&amp;rsquo;m hoping the following lessons
learned and observations will be useful to someone else and not simply cathartic.&lt;/p&gt;
&lt;h1 id=&#34;architecture-and-deployment&#34;&gt;Architecture and Deployment&lt;/h1&gt;
&lt;p&gt;Graylog2 can get really goddamn complicated. I kept referencing back to the &lt;a href=&#34;http://graylog2.org/resources/documentation/general/architecture&#34;&gt;Architecture&lt;/a&gt;
diagram as examples of what to design against and think about. I wasn&amp;rsquo;t trying to build something
that legends would be told of but I also didn&amp;rsquo;t want to build something that would have to be
ripped out and replaced in a year. So I spend a fair bit of time kidnapping a coworker and
scribbling on whiteboards as we discussed my ideas, though honestly I probably talked &lt;em&gt;at&lt;/em&gt; more
than &lt;em&gt;with&lt;/em&gt; him.&lt;/p&gt;
&lt;h2 id=&#34;graylog2-is-pretty-simple&#34;&gt;Graylog2 Is Pretty Simple&lt;/h2&gt;
&lt;p&gt;The guys at TORCH were either smart or lucky when they split up their architecture the way they
did. During my work the easiest component in the entire stack to scale out was the Graylog2 server
itself. There was no magic involved in creating a new server instance and expand a cluster. So long
as all of the &lt;code&gt;graylog2-server&lt;/code&gt; instances were configured against the same MongoDB server they were
clustered.&lt;/p&gt;
&lt;p&gt;Graylog2 itself doesn&amp;rsquo;t really need to scale because the application itself is pretty standalone.
There is the idea of a Graylog2 Cluster but as near as I can tell what that really means is, &amp;ldquo;I
see other &lt;code&gt;graylog2-server&lt;/code&gt; instances that say they&amp;rsquo;re part of me because they&amp;rsquo;re listed as online
in the database. That&amp;rsquo;s neato.&amp;rdquo; More than anything else it provides a way for more than one system
to write to the same ElasticSearch cluster without it turning into a cluster.&lt;/p&gt;
&lt;h2 id=&#34;logs-is-logs-is-logs&#34;&gt;Logs Is Logs Is Logs&lt;/h2&gt;
&lt;p&gt;Coming from an enterprise SIEM background I immediately jump to concepts like &amp;ldquo;log normalization&amp;rdquo;
and &amp;ldquo;dynamic baselineing&amp;rdquo;. These are features that we don&amp;rsquo;t really have here, and quite honestly I
shouldn&amp;rsquo;t complain. Graylog2 is primarily a log collection, storage, and search tool. Building out
dashboards with automatic dynamic baselines for each dashboard query is pretty system intensive
and quite a significant development effort to build and maintain. That&amp;rsquo;s not really the market this
product is going for.&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a href=&#34;#fn:2&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;p&gt;The other piece I desperately miss is log normalization. That is, every incoming log is run through
a parser and the important pieces of information are pulled out and stored in a consistent format.
This means we can do a search for &amp;ldquo;Authentication Logs for user Scott Pack&amp;rdquo; or &amp;ldquo;Show me all
successful logins for Scott Pack summarized by number of events per device&amp;rdquo; and receive consistent
results whether the logs came from Linux, Windows, or Apache. There is something called an
&amp;ldquo;Extractor&amp;rdquo; which is a start, but we&amp;rsquo;re not quite there yet. It&amp;rsquo;s an exceptionally heavy burden
to place on development staff so there&amp;rsquo;s always the possibility that normalization stays off the
radar.&lt;/p&gt;
&lt;h2 id=&#34;elasticsearch-is-magic&#34;&gt;ElasticSearch is Magic&lt;/h2&gt;
&lt;p&gt;For my complaints about MongoDB as a hipster software, ElasticSearch does lots of things right.
Clustering is trivial, one simply tells each node where the others are and they figure it out. My
only problems came into play when initially spreading out the load and then later when I performed
a major upgrade.&lt;/p&gt;
&lt;p&gt;Adding a second instance and setting them up in a Master/Master configuration was trivial, but by
default that only provides load sharing. I also had to change the Replication settings in order
for a copy of the data to exist on each node. That&amp;rsquo;s no big deal, but replication isn&amp;rsquo;t really set
at the global level. Instead every time an index is created the replication factor for that index
is set based on the global setting. This means I had to retroactively change the replication on
every index after the fact. ElasticSearch offers no direct method to make that change so I had to
write a script to iterate through every index and update the setting. Not terrible, but certainly
not great.&lt;/p&gt;
&lt;p&gt;The upgrade issue was a bit more interesting. Apparently, there is a disk utilization threshold
where if the partition that stores the ElasticSearch data is too full no shards will be assigned
to it. When I upgraded ElasticSearch from 0.90 to 1.3.2 my ElasticSearch usage was around 90% which
resulted in the second cluster node connecting to the cluster but never re-assigning its own
shards.&lt;/p&gt;
&lt;h1 id=&#34;management&#34;&gt;Management&lt;/h1&gt;
&lt;h2 id=&#34;yes-thats-right-i-said-plural-databases&#34;&gt;Yes. That&amp;rsquo;s Right. I Said Plural Databases.&lt;/h2&gt;
&lt;p&gt;Graylog2 uses two separate database systems. MongoDB stores configuration data and such from
&lt;code&gt;graylog2-web&lt;/code&gt; and &lt;code&gt;graylog2-server&lt;/code&gt; whereas ElasticSearch stores actual log data. Both are these
new fangled NoSQL JSON based RESTful systems. I&amp;rsquo;ll accept that the systems are actually designed
for different purposes. MongoDB acts more like a traditional database whereas ElasticSearch is
intended more as a data search engine. Using them for their intended purpose makes sense but
MongoDB is actually used to store a significantly insignificant amount of information. It makes
me wonder if, given the small amount of data involved, we couldn&amp;rsquo;t simply use ElasticSearch
instead.&lt;/p&gt;
&lt;p&gt;As it stands we have two separate database server technologies that use two separate management
subsystems. Which means my team needs to become experts in two more databases that aren&amp;rsquo;t used for
any other service that we manage. That problem lies solely on us, but it&amp;rsquo;s still a bummer.&lt;/p&gt;
&lt;h2 id=&#34;mongodb-makes-me-sad&#34;&gt;MongoDB Makes Me Sad&lt;/h2&gt;
&lt;p&gt;MongoDB provides some pretty amazing clustering and replication options. As I was splitting up
services and building configs I really thought I could &lt;a href=&#34;http://secopsmonkey.com/migrating-graylog2-servers-part-4.md&#34;&gt;build out a Master/Master cluster&lt;/a&gt; and
be happy. This is pretty much the anti-truth. In MongoDB a cluster is called a ReplicaSet, using
ReplicaSets we can build the distributed model we hope for but with an extra hitch called Quorum.
The &lt;a href=&#34;http://docs.mongodb.org/manual/core/replication-introduction/&#34;&gt;MongoDB Manual&lt;/a&gt; goes into more detail, but in short any ReplicaSet cluster requires at
minimum 3 nodes. Personally, I find 3 separate servers a bit much to store less than 5GB worth
of metadata.&lt;/p&gt;
&lt;h2 id=&#34;infinitely-unbounded-spoolers-are-the-devil&#34;&gt;Infinitely Unbounded Spoolers Are The Devil&lt;/h2&gt;
&lt;p&gt;As of this writing there exists a bug in &lt;code&gt;graylog2-server&lt;/code&gt; that results in a resource exhaustion
condition. The input message spooler caches files out to disk as they&amp;rsquo;re processed, which is cool.
What&amp;rsquo;s not cool is that the messages in the cache aren&amp;rsquo;t flushed as they&amp;rsquo;re processed. This means
the filesystem on which &lt;code&gt;/var/lib/graylog2-server/message-cache-spool&lt;/code&gt; resides will eventually
fill up and &lt;code&gt;graylog2-server&lt;/code&gt; will crash. Until the bug gets fixed, targeted for 0.92, my only
recourse was to set a cronjob that stops the service, deletes the cache files, and restarts the
service.&lt;/p&gt;
&lt;h1 id=&#34;support&#34;&gt;Support&lt;/h1&gt;
&lt;p&gt;In my experience having a good relationship with your providers is a necessity. It gives you a
voice to drive the product roadmap in a direction that helps your business. It gives you a
sympathetic ear that may be able to help when things go wrong. It gives you the chance to tell
someone who matters that you like what they&amp;rsquo;re doing and hope they&amp;rsquo;ll succeed. During the project
I engaged with some of the developers several times. Mostly to ask questions that came up, but
also to get the opportunity to become engaged and let them know what I wanted.&lt;/p&gt;
&lt;p&gt;The developers I interacted with were all pleasant people who really seemed to want to make a good
product and genuinely liked talking to people. I love this in a vendor. The best example is probably
this &lt;a href=&#34;https://twitter.com/graylog2/status/530367945009278976&#34;&gt;conversation&lt;/a&gt; on Twitter after they announced one of my most pined for features.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;Full Disclosure. The lead developer served as a guest on a coworker&amp;rsquo;s podcast. That happened
prior to my coming on board and as such didn&amp;rsquo;t involve me.&amp;#160;&lt;a href=&#34;#fnref:1&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;That being said if they &lt;em&gt;want&lt;/em&gt; to start digging that hole it&amp;rsquo;s be pretty awesome.&amp;#160;&lt;a href=&#34;#fnref:2&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>Migrating Graylog Servers - Part 5</title>
      <link>https://secopsmonkey.com/post/2014-10-31-migrating-graylog2-servers-part-5/</link>
      <pubDate>Fri, 31 Oct 2014 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-10-31-migrating-graylog2-servers-part-5/</guid>
      
        <description>&lt;p&gt;&lt;em&gt;This is the fifth in a multi-part series where I explore the process of transforming an existing
Graylog install into a resilient and scalable multi-site installation. &lt;a href=&#34;http://secopsmonkey.com/migrating-graylog2-servers.html&#34;&gt;Start here for Part 1&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;At this point we have the dedicated ElasticSearch nodes up and running with the legacy node shut
down. MongoDB has been split out and is in a Master/Slave replication mode running on the new
nodes. Graylog2 Server is up and running on the new instances with collection inputs created.
Graylog2 Web is running on all Graylog2 nodes. By now the only thing that should be left running
on the legacy node is the primary Graylog2 Web instance and the log listener on the legacy Graylog2
server.&lt;/p&gt;
&lt;p&gt;The final architectural piece to all this madness is to down the legacy Graylog2 instance and re-IP
its replacement. Since all of the syslog devices ship their logs to the legacy server this will let
us continue to receive logs while we rebalance all the end points.&lt;/p&gt;
&lt;p&gt;Our primary goal here is to make sure all of the configs are up to date before we re-IP the server
and then restart all the services.&lt;/p&gt;
&lt;h1 id=&#34;updating-graylog2-server&#34;&gt;Updating Graylog2 Server&lt;/h1&gt;
&lt;p&gt;On &lt;code&gt;east-gray01.east.example.com&lt;/code&gt; edit  &lt;code&gt;/etc/graylog2.conf&lt;/code&gt; and confirm the settings:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-yaml&#34; data-lang=&#34;yaml&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;l&#34;&gt;is_master = false&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;l&#34;&gt;elasticsearch_node_name = east-gray01&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;l&#34;&gt;elasticsearch_discovery_zen_ping_unicast_hosts = east-es01.east.example.com:9300,west-es01.west.example.com:9300&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;l&#34;&gt;rest_listen_uri = http://${IP_ADDRESS}:12900/&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Since &lt;code&gt;west-gray01&lt;/code&gt; is replacing the legacy server we&amp;rsquo;ll make it the master Graylog2 instance&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-yaml&#34; data-lang=&#34;yaml&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;l&#34;&gt;is_master = true&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;l&#34;&gt;elasticsearch_node_name = west-gray01&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;l&#34;&gt;elasticsearch_discovery_zen_ping_unicast_hosts = east-es01.east.example.com:9300,west-es01.west.example.com:9300&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;l&#34;&gt;rest_listen_uri = http://${IP_ADDRESS}:12900/&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;On both &lt;code&gt;east-gray01&lt;/code&gt; and &lt;code&gt;west-gray01&lt;/code&gt; edit &lt;code&gt;/etc/graylog2/web/graylog2-web-interface.conf&lt;/code&gt; and
make sure it points at the two new Graylog2 servers.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-yaml&#34; data-lang=&#34;yaml&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;l&#34;&gt;graylog2-server.uris=&amp;#34;http://east-es01.east.example.com:12900/,http://west-es01.west.example.com:12900/&amp;#34;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;h1 id=&#34;updating-elasticsearch&#34;&gt;Updating ElasticSearch&lt;/h1&gt;
&lt;p&gt;The ElasticSearch nodes should be mostly configured correctly already. For the most part we just
want to make sure the legacy node is removed. On both &lt;code&gt;east-es01&lt;/code&gt; and &lt;code&gt;west-es01&lt;/code&gt; make sure
&lt;code&gt;/etc/elasticsearch/elasticsearch.yml&lt;/code&gt; contains&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-yaml&#34; data-lang=&#34;yaml&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;node.master&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;true&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;node.data&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;true&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;discovery.zen.ping.multicast.enabled&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;false&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;discovery.zen.ping.unicast.hosts&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;east-es01.east.example.com&amp;#34;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;west-es01.west.example.com&amp;#34;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;]&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>Migrating Graylog Servers - Part 4</title>
      <link>https://secopsmonkey.com/post/2014-10-24-migrating-graylog2-servers-part-4/</link>
      <pubDate>Fri, 24 Oct 2014 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-10-24-migrating-graylog2-servers-part-4/</guid>
      
        <description>&lt;p&gt;&lt;em&gt;This is the fourth in a multi-part series where I explore the process of transforming an existing
Graylog install into a resilient and scalable multi-site installation. &lt;a href=&#34;http://secopsmonkey.com/migrating-graylog2-servers.html&#34;&gt;Start here for Part 1&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The MongoDB section actually gave me the biggest set of problems. Part of this is because of my
utter lack of familiarity with Mongo and partially because of Mongo&amp;rsquo;s design. Originally I hoped to
go for a full Master/Master fully replicated environment. This would meet the fully redundant
design that I was aiming for but during configuration I ran into a bit of snag. By design MongoDB
requires a minimum of 3 nodes so as not to result in hung juries during master elections. This
means that for my fully redundant geographically diverse setup I would need a minimum of 6 Mongo
nodes, 3 for east coast and 3 for west coast.&lt;/p&gt;
&lt;p&gt;Personally, I find the idea of running 6 servers to store indices, metadata, and configuration
data a bit much. Particularly considering the rest of the environment is planned to be a total of
4-6 systems. Due to this we opted to scale back a bit and go for Master/Slave replication. This
still gives us site diversity, and a backup node, but requires manual effort to promote the slave
to master.&lt;/p&gt;
&lt;p&gt;The process of promoting a single MongoDB instance to a master and adding a slave is actually very
straightforward. The basic workflow here is:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Convert the legacy standalone to a master node&lt;/li&gt;
&lt;li&gt;Add a new &lt;code&gt;mongod&lt;/code&gt; instance as a slave&lt;/li&gt;
&lt;li&gt;Wait for replication to occur&lt;/li&gt;
&lt;li&gt;Shutdown the &lt;code&gt;mongod&lt;/code&gt; instance on legacy standalone&lt;/li&gt;
&lt;li&gt;Promote new slave to master&lt;/li&gt;
&lt;li&gt;Add the other new &lt;code&gt;mongod&lt;/code&gt; instance as slave&lt;/li&gt;
&lt;li&gt;Wait for replication to occur&lt;/li&gt;
&lt;li&gt;Reconfigure &lt;code&gt;graylog2-server&lt;/code&gt; and &lt;code&gt;graylog2-web-server&lt;/code&gt; instances to use new Mongo nodes.&lt;/li&gt;
&lt;/ol&gt;
&lt;h1 id=&#34;convert-to-masterslave-and-replicate&#34;&gt;Convert to Master/Slave and Replicate&lt;/h1&gt;
&lt;p&gt;In order to replicate application data across we have to set up our legacy system to listen on a
non-localhost address, and to listen for slave connections. The configs are actually pretty easy.&lt;/p&gt;
&lt;p&gt;The biggest caveat to keep in mind is that the Graylog2 Server standalone instance will be
configured to hit mongodb on localhost. During the replication and promotion Graylog2 will not
be able to access the database instance.&lt;/p&gt;
&lt;p&gt;First edit &lt;code&gt;/etc/mongodb.conf&lt;/code&gt; on the legacy master&lt;/p&gt;
&lt;h2 id=&#34;legacy-master&#34;&gt;Legacy Master&lt;/h2&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;dbpath&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;sr&#34;&gt;/var/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;lib&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;mongodb&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;logpath&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;sr&#34;&gt;/var/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;log&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;mongodb&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;mongodb&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;log&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;logappend&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kp&#34;&gt;true&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;bind_ip&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;err&#34;&gt;$&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;public_ip&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nb&#34;&gt;fork&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;kp&#34;&gt;true&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;master&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;kp&#34;&gt;true&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Then edit &lt;code&gt;/etc/mongodb.conf&lt;/code&gt; on the new host&lt;/p&gt;
&lt;h2 id=&#34;new-slave&#34;&gt;New Slave&lt;/h2&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;dbpath&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;sr&#34;&gt;/var/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;lib&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;mongodb&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;logpath&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;sr&#34;&gt;/var/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;log&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;mongodb&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;mongodb&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;log&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;logappend&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kp&#34;&gt;true&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;bind_ip&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;err&#34;&gt;$&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;public_ip&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;journal&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kp&#34;&gt;true&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;slave&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;kp&#34;&gt;true&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;source&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;err&#34;&gt;$&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;master_node_address&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;h2 id=&#34;replication-times&#34;&gt;Replication Times&lt;/h2&gt;
&lt;p&gt;Since very little information is actually stored in Mongo replication should happen fairly fast.
We can monitor this on the filesystem by keeping track of the files stored in &lt;code&gt;/var/lib/mongodb&lt;/code&gt;.
Each database will, almost definitely, consume multiple files of the form &lt;code&gt;graylog2.#&lt;/code&gt;. Ignore the
files starting with &lt;code&gt;local&lt;/code&gt;, those are instance specific. Once the files settle down the database
will have finished replicating and we can move on.&lt;/p&gt;
&lt;h1 id=&#34;slave-promotion&#34;&gt;Slave Promotion&lt;/h1&gt;
&lt;p&gt;When it comes time to actually fiddle with the master node assignment we have several options to
choose from. Since my plan is to add yet another mongodb instance I opted to go with the &lt;a href=&#34;http://docs.mongodb.org/manual/core/master-slave/#failing-over-to-a-slave-promotion&#34;&gt;Failing
over to a slave (Promotion)&lt;/a&gt;. In a larger environment performing an inversion would allow us to
bring all the nodes into the replication cluster before removing the legacy system, but in my use
case it was just as easy to kill off the legacy box at the beginning.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Stop the mongodb instance on both master and slave: &lt;code&gt;initctl stop mongodb&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Edit the &lt;code&gt;/etc/mongodb.conf&lt;/code&gt; file on the new node setting it as the master
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;dbpath&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;sr&#34;&gt;/var/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;lib&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;mongodb&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;logpath&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;sr&#34;&gt;/var/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;log&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;mongodb&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;mongodb&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;log&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;logappend&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kp&#34;&gt;true&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;bind_ip&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;err&#34;&gt;$&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;public_ip&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nb&#34;&gt;fork&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;kp&#34;&gt;true&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;master&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;kp&#34;&gt;true&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/li&gt;
&lt;li&gt;Backup the &lt;code&gt;local.#&lt;/code&gt; files: &lt;code&gt;mkdir /var/lib/mongodb_local_bak &amp;amp;&amp;amp; mv /var/lib/mongodb/local* /var/lib/mongodb_local_bak/.&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Start up the mongodb instance on the new master: &lt;code&gt;initctl stop mongodb&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Restart the Graylog2 Server instances to connect to the new mongo instance: &lt;code&gt;initctl restart graylog2-server&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;h1 id=&#34;add-new-mongodb-instance-as-slave&#34;&gt;Add New MongoDB Instance as Slave&lt;/h1&gt;
&lt;p&gt;Now that the old master is out of the way we can add our new secondary as a slave. This is
done just like before&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;dbpath&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;sr&#34;&gt;/var/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;lib&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;mongodb&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;logpath&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;sr&#34;&gt;/var/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;log&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;mongodb&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;mongodb&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;log&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;logappend&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kp&#34;&gt;true&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;bind_ip&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;err&#34;&gt;$&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;public_ip&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;journal&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;kp&#34;&gt;true&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;slave&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;kp&#34;&gt;true&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;source&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;err&#34;&gt;$&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;new_master_node_address&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Once we&amp;rsquo;ve made these changes we can start &lt;code&gt;mongod&lt;/code&gt; on the new slave and let replication happen.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>Migrating Graylog Servers - Part 3</title>
      <link>https://secopsmonkey.com/post/2014-10-20-migrating-graylog2-servers-part-3/</link>
      <pubDate>Mon, 20 Oct 2014 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-10-20-migrating-graylog2-servers-part-3/</guid>
      
        <description>&lt;p&gt;&lt;em&gt;This is the third in a multi-part series where I explore the process of transforming an existing
Graylog install into a resilient and scalable multi-site installation. &lt;a href=&#34;http://secopsmonkey.com/migrating-graylog2-servers.html&#34;&gt;Start here for Part 1&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Previously we built our servers and &lt;a href=&#34;http://secopsmonkey.com/migrating-graylog2-servers-part-2.md&#34;&gt;reconfigured ElasticSearch&lt;/a&gt;. Next up is to build out the new
Graylog2 and Graylog2-Web servers themselves.&lt;/p&gt;
&lt;h1 id=&#34;graylog2-server-build&#34;&gt;Graylog2 Server Build&lt;/h1&gt;
&lt;h2 id=&#34;software-install&#34;&gt;Software Install&lt;/h2&gt;
&lt;p&gt;Since I sized and installed the OS instance at the same time as the ElasticSearch nodes I can jump
straight to the software install. In my example I&amp;rsquo;m using Ubuntu Server 14.10 LTS&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a href=&#34;#fn:1&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;. Since the
last time I set this stuff up Torch has started hosting their own software repos, which makes me
happy. We can manually set up and install the software by running&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nb&#34;&gt;echo&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;#34;deb http://packages.graylog2.org/repo/debian/ trusty 0.90&amp;#34;&lt;/span&gt; &amp;gt; /etc/apt/sources.list.d/graylog2.list
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;https://raw.githubusercontent.com/Graylog2/graylog2-puppet/master/files/RPM-GPG-KEY-graylog2
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;wget -qO - https://raw.githubusercontent.com/Graylog2/graylog2-puppet/master/files/RPM-GPG-KEY-graylog2 &lt;span class=&#34;p&#34;&gt;|&lt;/span&gt; apt-key add -
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;apt-get update
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;apt-get install graylog2-server graylog2-web&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Or if you want to use their &lt;a href=&#34;http://www.graylog2.org/resources/documentation/general/packages&#34;&gt;repo package&lt;/a&gt;&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;wget https://packages.graylog2.org/repo/packages/graylog2-0.91-repository-ubuntu14.04_latest.deb
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;dpkg -i graylog2-0.91-repository-ubuntu14.04_latest.deb
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;apt-get install apt-transport-https
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;apt-get update
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;apt-get install graylog2-server graylog2-web&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;h2 id=&#34;graylog2-cluster-join&#34;&gt;Graylog2 Cluster Join&lt;/h2&gt;
&lt;p&gt;Actually joining the Graylog2 instances to the cluster is pretty brain-dead easy as cluster
membership happens using the MongoDB instance. However, unlike ElasticSearch there are a &lt;em&gt;lot&lt;/em&gt;
of changes we need to make to the configuration file. The settings I&amp;rsquo;ve listed below are the
minimums. Please don&amp;rsquo;t entirely replace your &lt;code&gt;/etc/graylog2.conf&lt;/code&gt; with this. You need to read
through the file and make at least these changes&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a href=&#34;#fn:2&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;is_master&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;kp&#34;&gt;false&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;password_secret&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;err&#34;&gt;$&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;no&#34;&gt;Password_Secret_From_Legacy_Server&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;root_password_sha2&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;err&#34;&gt;$&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;no&#34;&gt;Password_Hash_From_Legacy_Server&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;rest_listen_uri&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;ss&#34;&gt;http&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;sr&#34;&gt;//&lt;/span&gt;&lt;span class=&#34;err&#34;&gt;$&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;no&#34;&gt;Public_IP_of_Server&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;}:&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;12900&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;retention_strategy&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;delete&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;elasticsearch_shards&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;mi&#34;&gt;5&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;elasticsearch_replicas&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;mi&#34;&gt;1&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;elasticsearch_index_prefix&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;graylog2&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;elasticsearch_cluster_name&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;err&#34;&gt;$&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;no&#34;&gt;ES_Cluster_Name&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;elasticsearch_node_name&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;err&#34;&gt;$&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;no&#34;&gt;Server_Hostname&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;elasticsearch_discovery_zen_ping_multicast_enabled&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;kp&#34;&gt;false&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;elasticsearch_discovery_zen_ping_unicast_hosts&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;east&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;-&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;es01&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;east&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;example&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;com&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;9300&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;west&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;-&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;es01&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;west&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;example&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;com&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;9300&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;graylog&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;example&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;com&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;9300&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;elasticsearch_cluster_discovery_timeout&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;mi&#34;&gt;10000&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;mongodb_useauth&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;kp&#34;&gt;false&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;mongodb_host&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;graylog&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;example&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;com&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;mongodb_database&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;graylog2&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;mongodb_port&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;mi&#34;&gt;27017&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;The most important things to notice here are&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;is_master&lt;/code&gt;: Unlike ElasticSearch this explicitly sets who is the master and like The Game &amp;ldquo;There can be only one.&amp;rdquo;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;rest_listen_uri&lt;/code&gt;: By default this is set to localhost, in order for the non-local graylog2-web instances to work this has to be externally reachable.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;elasticsearch_cluster_discovery_timeout&lt;/code&gt;: I made this number big since trans-continental links are relatively high latency.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;mongodb_host&lt;/code&gt;: Points at the MongoDB master, currently the legacy server.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;At this point we can start the &lt;code&gt;graylog2-server&lt;/code&gt; service and it should be pretty automagic. It will
auto-join the graylog2-server cluster based on data in the MongoDB instance. It will also join the
ElasticSearch cluster as a client node.&lt;/p&gt;
&lt;h2 id=&#34;graylog2-web-setup&#34;&gt;Graylog2 Web Setup&lt;/h2&gt;
&lt;p&gt;The Graylog2 Web config file is significantly shorter, and easier to deal with, than the Server
itself. Edit the file &lt;code&gt;/etc/graylog2/web/graylog2-web-interface.conf&lt;/code&gt; and make sure the following
are set.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;graylog2&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;-&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;server&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;uris&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;http://east-es01.east.example.com:12900/,http://west-es01.west.example.com:12900/,http://graylog.example.com:12900&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;application&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;secret&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;${Password_Secret_From_Legacy_Server}&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;timeout&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;.&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;DEFAULT&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;10&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;s&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Now start up the &lt;code&gt;graylog2-web&lt;/code&gt; service and we should be good to go. Note that we don&amp;rsquo;t actually
point the web service at either the ElasticSearch cluster or the Mongo database. All of the
information shown in the web interface comes through Graylog2 Server, including user accounts. The
data is still stored in the Mongo database it&amp;rsquo;s just not directly accessed by the web app.&lt;/p&gt;
&lt;p&gt;Now we have our new server and web interfaces up. Huzzah! At this point nothing is actually &lt;em&gt;using&lt;/em&gt;
the Graylog2 Servers, though, since all of the logs are still being shipped to the legacy system.
None of that will happen until after we retire the legacy server and start migrating hosts.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;Because DevOps.&amp;#160;&lt;a href=&#34;#fnref:1&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;The point of this blog is for me to document my process and to &lt;em&gt;hope&lt;/em&gt; that others can learn
from my mistakes and can start their project with at least more information than I did when I
started mine. The ability to copy/paste is not a substitute for a basic understanding of what the
fuck is going on.&amp;#160;&lt;a href=&#34;#fnref:2&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>My Not So Fancy .vimrc</title>
      <link>https://secopsmonkey.com/post/2014-10-10-my-not-so-fancy-vimrc/</link>
      <pubDate>Fri, 10 Oct 2014 00:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-10-10-my-not-so-fancy-vimrc/</guid>
      
        <description>&lt;p&gt;Sometime in the early 1990s, for reasons I can&amp;rsquo;t remember, I made the switch from pico to vim and
never looked back. Since that time I&amp;rsquo;ve been slowly tweaking my config, occasionally adding a
feature here and there, and always carrying my files along from one computer to another. My config
is still mostly stock, unlike some friends who read the &lt;a href=&#34;http://shop.oreilly.com/product/9780596529833.do&#34;&gt;O&amp;rsquo;Reilly&lt;/a&gt; book and somehow managed to
create a terminal version of Visual Studio. Every now and then I&amp;rsquo;ll be working on a project and
decide to look for a small boost. Something like syntax highlighting for LaTeX or puppet-lint for
live syntax validation on Puppet manifests.&lt;/p&gt;
&lt;p&gt;Every now and then I&amp;rsquo;ll have someone come along who starts getting into vim. I&amp;rsquo;ve been using vim
for so long that it&amp;rsquo;s a part of me so I always feel a little thrill when someone new rediscovers
something so old. I got this thrill just a few weeks ago when one of my Tweeples made the jump
completely unexpectedly. Unlike GUI based applications the full power of something like vim is
hard to wrap your head around. It&amp;rsquo;s all bound up in exceptionally dense and esoteric documentation
that can require an large amounts of organic experience to decode.&lt;/p&gt;
&lt;p&gt;Knowing this I like to make it easy for people. I try to dole out tips and tricks as they use it
so as not to become overwhelming. Which leads me to this post. Hoping it helps somebody scroll
down a bit and find the &lt;code&gt;.vimrc&lt;/code&gt; that I&amp;rsquo;ve been using all this time. Unlike what I did with my
&lt;a href=&#34;http://secopsmonkey.com/my-not-so-fancy-screenrc.html&#34;&gt;screenrc&lt;/a&gt; the &lt;code&gt;.vimrc&lt;/code&gt; is much too large to chunk and describe in detail. With any luck the
comments are sufficient to describe most of the settings.&lt;/p&gt;
&lt;p&gt;Probably the most non-obvious portion is the first bit referring to Pathogen. &lt;a href=&#34;http://www.vim.org/scripts/script.php?script_id=2332&#34;&gt;Pathogen&lt;/a&gt; is a
vim script that acts like a package manager for other vim scripts. Using the &lt;code&gt;infect&lt;/code&gt; method it
allows us to import all of the scripts we&amp;rsquo;ve installed in &lt;code&gt;~/.vim/bundle&lt;/code&gt; in one go. Specifically,
I&amp;rsquo;m loading:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Syntastic syntax checking plugin framework: &lt;a href=&#34;https://github.com/scrooloose/syntastic.git&#34;&gt;https://github.com/scrooloose/syntastic.git&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Puppet Syntax Highlighting and Style Checking: &lt;a href=&#34;https://github.com/rodjek/vim-puppet&#34;&gt;https://github.com/rodjek/vim-puppet&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://ethanschoonover.com/solarized&#34;&gt;Solarized&lt;/a&gt; color scheme: git://github.com/altercation/vim-colors-solarized.git&lt;/li&gt;
&lt;li&gt;LaTeX helper functions and highlighting: git://git.code.sf.net/p/vim-latex/vim-latex&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I find the most useful to be Syntastic. Whenever a file is written out it is syntax checked based
on the Syntastic plugin and any stylistic deviations or syntax warnings and errors are indicated
in a sidebar.&lt;/p&gt;
&lt;p&gt;Setting it all up is pretty easy. Just follow the instructions on installing Pathogen and drop the
following in your &lt;code&gt;~/.vimrc&lt;/code&gt;.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-vim&#34; data-lang=&#34;vim&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Setup the environment for Syntastic syntax checking&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nx&#34;&gt;call&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;pathogen&lt;/span&gt;#&lt;span class=&#34;nx&#34;&gt;infect&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;s1&#34;&gt;&amp;#39;bundle/{}&amp;#39;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Enable syntax highlighting&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;syntax&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;on&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Show text description of code warnings and Syntastic notices in status line&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;statusline&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;+=&lt;/span&gt;%#&lt;span class=&#34;nx&#34;&gt;warningmsg&lt;/span&gt;#
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;statusline&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;+=&lt;/span&gt;%{&lt;span class=&#34;nx&#34;&gt;SyntasticStatuslineFlag&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;()&lt;/span&gt;}
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;statusline&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;+=&lt;/span&gt;%*
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt; 
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;filetype&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;plugin&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;on&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;filetype&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;indent&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;on&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Set up some settings for vim-latex&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;let&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;g&lt;/span&gt;:&lt;span class=&#34;nx&#34;&gt;tex_flavor&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s1&#34;&gt;&amp;#39;latex&amp;#39;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Disable all that folding nonsense&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;let&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;Tex_FoldedSections&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;let&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;Tex_FoldedEnvironments&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;let&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;Tex_FoldedMisc&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Set up the Solarized colorscheme&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;background&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;nb&#34;&gt;dark&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;let&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;g&lt;/span&gt;:&lt;span class=&#34;nx&#34;&gt;solarized_visibility&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;#34;high&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;let&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;g&lt;/span&gt;:&lt;span class=&#34;nx&#34;&gt;solarized_contrast&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;#34;high&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;let&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;g&lt;/span&gt;:&lt;span class=&#34;nx&#34;&gt;solarized_termcolors&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;m&#34;&gt;256&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;let&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;g&lt;/span&gt;:&lt;span class=&#34;nx&#34;&gt;solarized_termtrans&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;m&#34;&gt;1&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;colorscheme&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;solarized&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;modeline&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Always assume paste mode&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;paste&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Enable Markdown formatting&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nx&#34;&gt;augroup&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;mkd&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;k&#34;&gt;autocmd&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;BufRead&lt;/span&gt; *.&lt;span class=&#34;nx&#34;&gt;mkd&lt;/span&gt;  &lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;ai&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;formatoptions&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;nx&#34;&gt;tcroqn2&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;comments&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;nx&#34;&gt;n&lt;/span&gt;:&amp;amp;&lt;span class=&#34;nx&#34;&gt;gt&lt;/span&gt;;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nx&#34;&gt;augroup&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;END&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Disable autoindenting&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;noai&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; have command-line completion &amp;lt;Tab&amp;gt; (for filenames, help topics, option names)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; first list the available options and complete the longest common part, then&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; have further &amp;lt;Tab&amp;gt;s cycle through the possibilities:&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;wildmode&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;nx&#34;&gt;list&lt;/span&gt;:&lt;span class=&#34;nx&#34;&gt;longest&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;nx&#34;&gt;full&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Turn the mouse off for all modes, graphical and console&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;mouse&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; When entering a tab with this on, spaces are entered instead (at 4 characters).&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; To enter a real tab, use  CTRL-V TAB.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;expandtab&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;tabstop&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;4&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;shiftwidth&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;4&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; make searches case-insensitive, unless they contain upper-case letters:&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;ignorecase&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;smartcase&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; show the `best match so far&amp;#39; as search strings are typed:&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;incsearch&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; show cursor line and column in the status line&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;ruler&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; show matching brackets&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;showmatch&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; display the current mode and partially-typed commands in the status line:&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;showmode&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;showcmd&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Allows full backspacing, including over insertion points and line breaks&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;bs&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;2&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Required to be able to use keypad keys and map missed escape sequences&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;esckeys&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Disable vi compatability&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;nocompatible&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; enable filetype detection:&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;filetype&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;on&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; Syntax highlighting for iCal ics files&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;autocmd&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;!&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;BufRead&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;nx&#34;&gt;BufNewFile&lt;/span&gt; *.&lt;span class=&#34;nx&#34;&gt;ics&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;setfiletype&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;icalendar&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; for C-like programming, have automatic indentation:&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;autocmd&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;FileType&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;c&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;nx&#34;&gt;cpp&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;nx&#34;&gt;slang&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;cindent&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;cinkeys&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;cino&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;1&lt;/span&gt;s
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; for actual C (not C++) programming where comments have explicit end&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; characters, if starting a new line in the middle of a comment automatically&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; insert the comment leader characters:&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;autocmd&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;FileType&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;c&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;formatoptions&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;+=&lt;/span&gt;&lt;span class=&#34;nx&#34;&gt;ro&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; for CSS, also have things in braces indented:&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;autocmd&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;FileType&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;css&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;smartindent&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; for HTML, generally format text, but if a long line has been created leave it&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; alone when editing:&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;autocmd&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;FileType&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;html&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;formatoptions&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;+=&lt;/span&gt;&lt;span class=&#34;nx&#34;&gt;tl&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; for both CSS and HTML, use genuine tab characters for indentation, to make&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; files a few bytes smaller:&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;autocmd&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;FileType&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;html&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;nx&#34;&gt;css&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;noexpandtab&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;tabstop&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;4&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; in makefiles, don&amp;#39;t expand tabs to spaces, since actual tab characters are&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; needed, and have indentation at 8 chars to be sure that all indents are tabs&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c&#34;&gt;&amp;#34; (despite the mappings later):&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;autocmd&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;FileType&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;make&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;set&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;noexpandtab&lt;/span&gt; &lt;span class=&#34;nx&#34;&gt;shiftwidth&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;m&#34;&gt;8&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>Migrating Graylog Servers - Part 2</title>
      <link>https://secopsmonkey.com/post/2014-10-03-migrating-graylog2-servers-part-2/</link>
      <pubDate>Fri, 03 Oct 2014 00:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-10-03-migrating-graylog2-servers-part-2/</guid>
      
        <description>&lt;p&gt;&lt;em&gt;This is the second in a multi-part series where I explore the process of transforming an existing
Graylog install into a resilient and scalable multi-site installation. &lt;a href=&#34;http://secopsmonkey.com/migrating-graylog2-servers.html&#34;&gt;Start here for Part 1&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;
&lt;h1 id=&#34;server-setup&#34;&gt;Server Setup&lt;/h1&gt;
&lt;p&gt;At this point we have about 6 months of data in our existing Graylog2 system so I wanted the entire
migration to happen without data loss and to be as seamless as possible. For the new environment I
needed 4 new servers:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;East Coast Graylog2 Server (east-gray01)&lt;/li&gt;
&lt;li&gt;East Coast ElasticSearch node (east-es01)&lt;/li&gt;
&lt;li&gt;West Coast Graylog2 Server (west-gray01)&lt;/li&gt;
&lt;li&gt;West Coast ElasticSearch node (west-es01)&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;You&amp;rsquo;ll notice a discrepancy between this setup and the diagram from before. For now I opted to leave
the Graylog2 Web Interface running on the Graylog2 server nodes. That was mostly for easy of
deployment and management more than anything else. I&amp;rsquo;m truly not concerned with load at this point,
and we will be configuring the applications with no location assumptions. This means if we need to
separate the two out then there will be minimal amounts of detangling involved.&lt;/p&gt;
&lt;p&gt;The &lt;a href=&#34;http://www.graylog2.org/resources/documentation&#34;&gt;Architectural Considerations&lt;/a&gt; gives us some pretty good data on sizing. Server nodes need
gobs of CPU, data nodes need gobs of RAM and disk but not much CPU, web and mongo need almost
pitifully small amounts of anything. As a general rule I would recommend the following for two
deployment scenarios.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Graylog2 Servers&lt;/strong&gt;&lt;/p&gt;
&lt;table&gt;
  &lt;thead&gt;
      &lt;tr&gt;
          &lt;th&gt;&lt;/th&gt;
          &lt;th&gt;Small&lt;/th&gt;
          &lt;th&gt;Medium&lt;/th&gt;
      &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
      &lt;tr&gt;
          &lt;td&gt;CPU&lt;/td&gt;
          &lt;td&gt;4 cores&lt;/td&gt;
          &lt;td&gt;8 cores&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;RAM&lt;/td&gt;
          &lt;td&gt;4 GB&lt;/td&gt;
          &lt;td&gt;8 GB&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;Disk&lt;/td&gt;
          &lt;td&gt;40 GB&lt;/td&gt;
          &lt;td&gt;40 TB&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;OS&lt;/td&gt;
          &lt;td&gt;Ubuntu 14.04&lt;/td&gt;
          &lt;td&gt;Ubuntu 14.04&lt;/td&gt;
      &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;&lt;strong&gt;ElasticSearch Servers&lt;/strong&gt;&lt;/p&gt;
&lt;table&gt;
  &lt;thead&gt;
      &lt;tr&gt;
          &lt;th&gt;&lt;/th&gt;
          &lt;th&gt;Small&lt;/th&gt;
          &lt;th&gt;Medium&lt;/th&gt;
      &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
      &lt;tr&gt;
          &lt;td&gt;CPU&lt;/td&gt;
          &lt;td&gt;2 cores&lt;/td&gt;
          &lt;td&gt;2 cores&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;RAM&lt;/td&gt;
          &lt;td&gt;12 GB&lt;/td&gt;
          &lt;td&gt;24 GB&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;Disk&lt;/td&gt;
          &lt;td&gt;500 GB&lt;/td&gt;
          &lt;td&gt;1 TB&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;OS&lt;/td&gt;
          &lt;td&gt;Ubuntu 14.04&lt;/td&gt;
          &lt;td&gt;Ubuntu 14.04&lt;/td&gt;
      &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Very likely the most up-in-the-air figure is going to be disk. If you are just now building a log
collector it&amp;rsquo;s hard to tell what the overall utilization is. You&amp;rsquo;ll need to take into account your
retention schedule, the typical size of your logs, and how quickly they come in.&lt;/p&gt;
&lt;p&gt;Actually building the servers is easy. These days we have fan-dancy tools like Foreman or Cobbler
and Puppet or Chef to do all the actual work leaving us with a base system to deal with. I opted
to get started on the ElasticSearch.&lt;/p&gt;
&lt;p&gt;Since site independence was foremost in my mind I wanted to to emulate a master-master replication
setup. This was actually significantly easier than I thought, largely due to the automagic
clustering built into ElasticSearch.&lt;/p&gt;
&lt;h1 id=&#34;elasticsearch-fiddly-bits&#34;&gt;ElasticSearch Fiddly Bits&lt;/h1&gt;
&lt;h2 id=&#34;installation&#34;&gt;Installation&lt;/h2&gt;
&lt;p&gt;First we have to install the ElasticSearch application. Graylog2 has a hard dependency on version
0.90.10&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a href=&#34;#fn:1&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;. Fortunately for us ElasticSearch maintains their own &lt;a href=&#34;http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-repositories.html&#34;&gt;package repositories&lt;/a&gt; based
on primary version. Due to the requirement of the older version we can adjust their instructions a
tad to be&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;wget -qO - http://packages.elasticsearch.org/GPG-KEY-elasticsearch &lt;span class=&#34;p&#34;&gt;|&lt;/span&gt; sudo apt-key add -
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nb&#34;&gt;echo&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;#34;deb http://packages.elasticsearch.org/elasticsearch/0.90/debian stable main&amp;#34;&lt;/span&gt; &amp;gt; /etc/apt/sources.list.d/elasticsearch.list
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;apt-get update &lt;span class=&#34;o&#34;&gt;&amp;amp;&amp;amp;&lt;/span&gt; apt-get install elasticsearch&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Keep in mind that will install the newest version of ElasticSearch from the 0.90 tree. Since I
do all the menial work with Puppet I fix the problem by doing this in a manifest&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;package&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;s1&#34;&gt;&amp;#39;elasticsearch&amp;#39;&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;k&#34;&gt;ensure&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&#34;s1&#34;&gt;&amp;#39;0.90.10&amp;#39;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;This makes sure the version 0.90.10 is always installed, no matter what the newest version is&lt;sup id=&#34;fnref:2&#34;&gt;&lt;a href=&#34;#fn:2&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;h2 id=&#34;build-out-the-application-configuration&#34;&gt;Build Out the Application Configuration&lt;/h2&gt;
&lt;p&gt;Once the application is installed we need to configure it to join our current cluster. Edit the
file &lt;code&gt;/etc/elasticsearch/elasticsearch.yml&lt;/code&gt; to look something like this.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-yaml&#34; data-lang=&#34;yaml&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;cluster.name&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;l&#34;&gt;graylog2&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;node.name&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;l&#34;&gt;east-es01&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;node.master&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;true&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;node.data&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;true&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;index.number_of_shards&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;m&#34;&gt;5&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;index.number_of_replicas&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;m&#34;&gt;1&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;discovery.zen.ping.timeout&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;l&#34;&gt;5s&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;discovery.zen.ping.multicast.enabled&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;kc&#34;&gt;false&lt;/span&gt;&lt;span class=&#34;w&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;nt&#34;&gt;discovery.zen.ping.unicast.hosts&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;:&lt;/span&gt;&lt;span class=&#34;w&#34;&gt; &lt;/span&gt;&lt;span class=&#34;p&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;east-es01.east.example.com&amp;#34;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;west-es01.west.example.com&amp;#34;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;graylog.example.com&amp;#34;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;]&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Some of the ElasticSearch terminology threw me for a bit, particularly the &lt;code&gt;node.master&lt;/code&gt; setting.
Unlike Graylog2, where you &lt;em&gt;define&lt;/em&gt; which node is the master, setting &lt;code&gt;node.master&lt;/code&gt; to true simply
indicates that this node &lt;em&gt;is capable&lt;/em&gt; of being a master. The ElasticSearch cluster itself elects a
master from the current member nodes. It&amp;rsquo;s much like a NetBIOS Browser Election except not nearly
as terrifyingly noisy.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;node.data&lt;/code&gt; says whether this system will actually store anything or act solely as a worker node&lt;/li&gt;
&lt;li&gt;&lt;code&gt;discovery.zen.ping.timeout&lt;/code&gt; defines how long the master election takes. The default is 3s but since I&amp;rsquo;m dealing with a cross-continent haul I was getting failures. Bumping it up to 5 fixed things.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;discovery.zen.ping.unicast.hosts&lt;/code&gt; is used to restrict cluster membership and should contain every other ES node in the cluster. I opted to include all of them since it doesn&amp;rsquo;t cause problems and it lets me reuse the same config.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;discovery.zen.ping.multicast.enabled&lt;/code&gt; is pretty important. By default ES uses multicast to determine other cluster members. I don&amp;rsquo;t like that since it&amp;rsquo;s noisy and lets anyone in the multicast domain gain access to your cluster.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;index.number_of_replicas&lt;/code&gt; defines how many copies of the data exist and is the real magic here.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Determining the number of shards per index is fuzzy. You want it bigger than one because the bigger
the index the more resources are consumed writing and indexing it, plus the bigger the shard the
longer it takes to replicate. Presumably there&amp;rsquo;s a corollary wherein an insane number of shards can
blow up performance. In our environment we did some rough envelope math to make each index roughly
4GB and went with 5 shards because it was halfway to 10. Setting replication to 1 means that every
shard will have exactly 1 copy, i.e. 1 primary and 1 replica. It will also distribute the replicas
in such a way as to minimize data duplication. That is, if we have 2 ElasticSearch nodes and 1
replica the primary will be on east-es01 and the replica will be on west-es01. You should only ever
have both a primary and replica on the same node if you have more replicas than nodes.&lt;/p&gt;
&lt;p&gt;If you make the right call and install from the repo they&amp;rsquo;ll be a service configuration file, for
us it&amp;rsquo;ll be &lt;code&gt;/etc/default/elasticsearch&lt;/code&gt;. Change the value of &lt;code&gt;ES_HEAP_SIZE&lt;/code&gt; to something much
bigger. This should be at least half of your total memory.&lt;/p&gt;
&lt;h2 id=&#34;make-it-happen&#34;&gt;Make It Happen&lt;/h2&gt;
&lt;p&gt;Initially we want data to start being replicated without the new boxes taking complete control. We
can do this by copying the above config out to the new nodes but changing &lt;code&gt;node.master&lt;/code&gt; to &lt;code&gt;false&lt;/code&gt;.
We&amp;rsquo;ll also need to change the &lt;code&gt;discovery.zen.ping.multicast.enabled&lt;/code&gt; and
&lt;code&gt;discovery.zen.ping.unicast.hosts&lt;/code&gt; so the new nodes can join the cluster. Once all the files get
copied into place start up all the services and magic should happen. You should start seeing shards
get reallocated between nodes as the cluster tries to balance itself out.&lt;/p&gt;
&lt;p&gt;Something to keep in mind is that the number of shards per index is fixed at index creation time
and cannot be changed. The number of replicas per shard, however, can be changed. While setting
the value in the config files only applies to new indices we can use the API to retroactively
adjust the replica values. The authoritative information on the how can be found in the &lt;a href=&#34;http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-update-settings.html&#34;&gt;Reference
Guide&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;There are a number of monitoring tools for ElasticSearch that will help you keep an eye on
replication status. I used &lt;a href=&#34;https://github.com/lmenezes/elasticsearch-kopf/&#34;&gt;kopf&lt;/a&gt; but the &lt;a href=&#34;http://www.elasticsearch.org/guide/en/elasticsearch/client/community/current/health.html&#34;&gt;Health and Performance Monitoring&lt;/a&gt; page lists a
few packages to look at.&lt;/p&gt;
&lt;p&gt;Once all the indices are updated and the replication is finished we can immediately remove the
original ElasticSearch instance from the cluster. However, for cleanliness, I&amp;rsquo;m going to wait until
the stack is built out before I start removing components.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;This is actually changing soon, for some definitions of soon. According to the &lt;a href=&#34;http://www.graylog2.org/news/post/0005-graylog2-v0-90-has-been-released&#34;&gt;release notes
for Graylog2 v0.90&lt;/a&gt; we have optional support for ElasticSearch v1.3.2 in Graylog2 v0.91-rc.1.
Don&amp;rsquo;t get confused by the numbers, suffice to say we can upgrade soon.&amp;#160;&lt;a href=&#34;#fnref:1&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;
&lt;p&gt;It&amp;rsquo;s worth noting that if you run an upgrade through apt-get this won&amp;rsquo;t prevent ElasticSearch
from being updated. I&amp;rsquo;m sure apt has a way to exclude packages based on version, though I&amp;rsquo;m also
sure the syntax is different from yum and something I should do sooner rather than later.&amp;#160;&lt;a href=&#34;#fnref:2&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>Migrating Graylog Servers</title>
      <link>https://secopsmonkey.com/post/2014-09-19-migrating-graylog2-servers/</link>
      <pubDate>Fri, 19 Sep 2014 00:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-09-19-migrating-graylog2-servers/</guid>
      
        <description>&lt;p&gt;&lt;em&gt;This is the first in a multi-part series where I explore the process of transforming an existing
Graylog install into a resilient and scalable multi-site installation.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;When I started at &lt;code&gt;${DayJob}&lt;/code&gt; we were using a single server Graylog2 for log storage. Honestly it
works out pretty well, the only real resource hog is ElasticSearch which will eat up as much
memory as you can ever throw at it. Occasionally we would notice the Java processes would be
running low on memory so we&amp;rsquo;d add a few more GB to the VM and walk away. Since that time there
have been several new versions of GL2 released, installation has gotten easier, and most importantly
we now have packages that install to predictable locations. Instead of trying to migrate an existing
system from a source install to package I opted to take the opportunity to fully evaluate the design
of our logging environment.&lt;/p&gt;
&lt;p&gt;My team supports a mix of Windows and Linux servers, almost entirely virtualized, and a large number
of network devices spread out among a fistful of remote office sites and multiple geographically
distributed datacenters. As one might expect certain business units are more heavily supported in
one location or another but whenever possible, and when it actually makes sense, we strive for
proper site diversity. I definitely wanted to take this into account during my evaluation.&lt;/p&gt;
&lt;p&gt;Quite frankly, the the single-server stack still provides sufficient performance so don&amp;rsquo;t need
anything quite so complex as the Bigger Production Setup described in the &lt;a href=&#34;http://graylog2.org/resources/documentation/general/architecture&#34;&gt;Architectural
Considerations&lt;/a&gt;. However we can somewhat easily adapt it to fit our needs. The overall goal
of the new design was to achieve something akin to a hot active-active environment with
cross-site replication. For performance purposes each site should have their own log collection
capabilities, similarly the Graylog2 Servers should store collected logs to a local ElasticSearch
data node. In order to tolerate the entire loss of a site each site should be able to operate
independently and, more importantly, we shouldn&amp;rsquo;t have to worry about data loss if a site
disappears or the link breaks. To that end we&amp;rsquo;re going to take a setup that looks something like
this&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-09-19-migrating-graylog2-servers/current_graylog2_design.png&#34;
    alt=&#34;Current Logging Design&#34;&gt;
&lt;/figure&gt;

&lt;p&gt;to one that&amp;rsquo;s more like&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-09-19-migrating-graylog2-servers/new_graylog2_design.png&#34;
    alt=&#34;New Fangled Logging Design&#34;&gt;
&lt;/figure&gt;

&lt;p&gt;Fortunately the underlying ElasticSearch environment has pretty good support for scaling out
additional nodes, as does Graylog2 Server. This will let us build and migrate into the new
environment pretty seamlessly, with only a minor downtime while we swap out the old monolithic
server with the new dedicated Graylog2 instance for that site. At a high level what we&amp;rsquo;ll need
to do is&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Build the new servers.&lt;/li&gt;
&lt;li&gt;Add the new ElasticSearch data nodes to the cluster and adjust sharding configurations.&lt;/li&gt;
&lt;li&gt;Add the new Graylog2 Server instances to the cluster.&lt;/li&gt;
&lt;li&gt;Add the new Graylog2 Web instances to the cluster.&lt;/li&gt;
&lt;li&gt;Swap the existing Graylog2 Server instance.&lt;/li&gt;
&lt;li&gt;Remove the existing Graylog2 Server instance from the cluster.&lt;/li&gt;
&lt;li&gt;Remove the existing ElasticSearch data node from the cluster.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Over the course of the next few posts I&amp;rsquo;ll be documenting the process I took, all of the surprises
that arose, and how I managed to beat each of them into submission.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>Resolving Conflicts Between SSH and Read Only Mounts</title>
      <link>https://secopsmonkey.com/post/2014-08-15-resolving-conflicts-between-ssh-and-read-only-mounts/</link>
      <pubDate>Fri, 15 Aug 2014 00:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-08-15-resolving-conflicts-between-ssh-and-read-only-mounts/</guid>
      
        <description>&lt;blockquote&gt;
&lt;p&gt;Pulling back from the archives this is a repost of a previous blog post. This time ripped from a
guest spot at &lt;a href=&#34;http://thenubbyadmin.com/2012/11/07/resolving-conflicts-between-ssh-and-read-only-mounts/&#34;&gt;The Nubby Admin&lt;/a&gt;, a fantastic blog from a fellow tech nerd.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;At my old job I had a bottom of the line box sitting on my desk that I used for
some testing. It was a hardware clone of the oldest Snort sensors I had
deployed, and by old I mean corporate desktop grade vintage 2003. I kept it
running and set up so that I can test configuration changes, new rules, software
updates, etc. This all makes the system pretty mundane and cookie cutter. I&amp;rsquo;d
often get a hankering to start from a blank slate, tell this thing to go koan
itself, and come back to a clean install. This is all a rather long and drawn
out way of saying that I didn&amp;rsquo;t really care about the health of this system and
didn&amp;rsquo;t pay very much attention to it.&lt;/p&gt;
&lt;p&gt;One morning, in the email containing the output from the auto-update script for
this system, I saw gobs of errors of the form:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;Error unpacking rpm package 4:perl-5.8.8-38.el5_8.i386
error: unpacking of archive failed on file /usr/bin/a2p;509097e4: cpio: open failed - Read-only file system
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;If there&amp;rsquo;s one way to catch my interest, it is to tell me that system partitions
are read-only. Some quick research showed me that yes, every single file-system
was in fact read only. SMART also was showing errors out the wazoo. Over the
years I have learned that SMART&amp;rsquo;s false-negative rate was astronomical, but its
false-positive rate was approaching zero. That is, a &amp;ldquo;healthy&amp;rdquo; report from SMART
was meaningless, but a &amp;ldquo;failed&amp;rdquo; report is completely trustworthy. This was
really no big deal, since the system wasn&amp;rsquo;t used for anything time sensitive, I
could just pull a hard drive off the shelf and reprovision.&lt;/p&gt;
&lt;p&gt;Since one thing I had used this system for was to do some performance profiling
of snort, I had mocked up a couple of analysis and test scripts on it. The
scripts themselves were easily rebuildable, but to save me the effort I used the
old tar+ssh trick to archive the home directories for myself and root for later
extraction:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;tar -zc /home/packs /root | ssh packs@node1 &#39;cat - &amp;gt; snort-test_homes.tar.gz&#39;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This is where things got hairy. Since the last time I had used ssh to go to
node1 it had also been rebuilt, resulting in a host key change. With
StrictHostKeyChecking enabled ssh refuses to connect if there is a host key
mismatch. Ordinarily, I would just delete the key from the &lt;code&gt;known_hosts&lt;/code&gt; file
and move on. With the file-system being read only&amp;hellip;.&lt;/p&gt;
&lt;p&gt;I worked around this by changing my &lt;code&gt;known_hosts&lt;/code&gt; file on the command line.
Since all the file-systems were read-only I couldn&amp;rsquo;t actually write any files,
nor did I need to save the information. This left me with the perfect choice of
&lt;code&gt;/dev/null&lt;/code&gt;. Adding in this option made my final command look something like
this:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;tar -zc /home/packs /root | ssh packs@node1 -o UserKnownHostsFile=/dev/null &#39;cat - &amp;gt; snort-test_homes.tar.gz&#39;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I liked this because it worked and was easy. I don&amp;rsquo;t like it because it felt skeezy.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>How I Learned to Tolerate White on Fuscia</title>
      <link>https://secopsmonkey.com/post/2014-08-08-how-i-learned-to-tolerate-white-on-fuscia/</link>
      <pubDate>Fri, 08 Aug 2014 00:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-08-08-how-i-learned-to-tolerate-white-on-fuscia/</guid>
      
        <description>&lt;blockquote&gt;
&lt;p&gt;Pulling back from the archives this is a repost of a previous blog post. This time ripped from a
guest spot at &lt;a href=&#34;http://thenubbyadmin.com/2012/04/03/how-i-learned-to-tolerate-white-on-fuscia/&#34;&gt;The Nubby Admin&lt;/a&gt;, a fantastic blog from a fellow tech nerd.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;One of the worst problems I have with tabbed consoles is knowing exactly which console I&amp;rsquo;m working
in. Sure, I can simply look at either the shell&amp;rsquo;s title, or the prompt, but I still inevitably will
type a command into the wrong console. Normally this will result in something more akin to &amp;ldquo;file not
found&amp;rdquo; because you catted the wrong file, rather than &amp;ldquo;Now running koan on all servers&amp;rdquo; because you
did something in mcollective prod instead of test. While chatting up a guy at work about this
problem we threw around changing the background color depending on which system you&amp;rsquo;re on.&lt;/p&gt;
&lt;p&gt;This is one of those things that&amp;rsquo;s a little trickier than one might think at first blush. For
instance, my first thought was to overload the ssh operator and make the changes client side. This
works for me since I use cygwin as my ssh client and proxy through an aggregator (see this
&lt;a href=&#34;http://serverfault.com/a/344553/3356&#34;&gt;ServerFault answer&lt;/a&gt; about proxying). My coworker, however, uses puttycm which is a fair
bit more restrictive and doesn&amp;rsquo;t allow me to do any kind of local overloading. Largely because of
this I was force to push the colorization to the remote host. Colorizing remotely had the advantage
of meeting both our use cases but at the expense of requiring changes on every single system we
might ssh into.&lt;/p&gt;
&lt;p&gt;While I proxy all of my sessions through the jump host he would first log into the jump host and
then remote elsewhere. This meant we had to change the background color on login, but then change
it back on logoff. Otherwise an accidentally reused session could have incorrect coloring. To that
end, I decided to make the colorization a separate function so it could easily be reused.&lt;/p&gt;
&lt;p&gt;It&amp;rsquo;s also worth noting that this was designed for a system running bash-3.x. I still deal with a lot
of RHEL5 systems, which uses bash-3.x, so I coded to it for maximum compatability. Since associative
arrays weren&amp;rsquo;t added until bash-4.x I had to do some odd stuff with the arrays that basically amount
to magic.&lt;/p&gt;
&lt;p&gt;Without further ado, below is everything you&amp;rsquo;ll need to add to your &lt;code&gt;~/.bashrc&lt;/code&gt; file to make this
garbage work. With any luck, the comments are sufficient to explain.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;function&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;setcolor&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Set up the colormap using hex codes for the colors&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;n&#34;&gt;colormap&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;#34;node1:#330000&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;             &lt;span class=&#34;s2&#34;&gt;&amp;#34;node2:#003300&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;             &lt;span class=&#34;s2&#34;&gt;&amp;#34;node3:#330033&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;             &lt;span class=&#34;s2&#34;&gt;&amp;#34;node4:#333300&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;             &lt;span class=&#34;s2&#34;&gt;&amp;#34;node5:#FF0000&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;             &lt;span class=&#34;s2&#34;&gt;&amp;#34;node6:#0000FF&amp;#34;&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Generate my own short hostname, i.e. turn node1.example.com into node1&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;n&#34;&gt;short&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;sb&#34;&gt;`echo ${HOSTNAME} | sed &amp;#34;s/..*$//&amp;#34;`&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;color&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;#000000&amp;#34;&lt;/span&gt; &lt;span class=&#34;c1&#34;&gt;# Set default color to black&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Iterate through the colormap looking for the hostname. Also, some bash magic.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;k&#34;&gt;for&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;host&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;in&lt;/span&gt; &lt;span class=&#34;err&#34;&gt;$&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;colormap&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;err&#34;&gt;@&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;};&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;do&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;k&#34;&gt;if&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;[[&lt;/span&gt; &lt;span class=&#34;err&#34;&gt;$&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;host&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;%&lt;/span&gt;&lt;span class=&#34;s&#34;&gt;%:*} == ${short} ]]; then
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;s&#34;&gt;      color=${host##*:&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;n&#34;&gt;fi&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;n&#34;&gt;done&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Wrap the color in the xterm escape sequences to set the background color&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;n&#34;&gt;echo&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;-&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;ne&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;#34;33]11;${color}07&amp;#34;&lt;/span&gt; &lt;span class=&#34;c1&#34;&gt;# Set the background color&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Only run the setcolor function if we are using xterm or xterm-color as our termtype&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;if&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;[&lt;/span&gt; &lt;span class=&#34;vg&#34;&gt;$TERM&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;#34;xterm&amp;#34;&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;then&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;n&#34;&gt;export&lt;/span&gt; &lt;span class=&#34;no&#34;&gt;TERM&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;s2&#34;&gt;&amp;#34;xterm-color&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;n&#34;&gt;setcolor&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;elif&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;[&lt;/span&gt; &lt;span class=&#34;vg&#34;&gt;$TERM&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;#34;xterm-color&amp;#34;&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;then&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;n&#34;&gt;setcolor&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;fi&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Replaces the shell title with the name of the host we are sshing to&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;function&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;ssh&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;echo&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;-&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;ne&lt;/span&gt; &lt;span class=&#34;s2&#34;&gt;&amp;#34;33]0;${1}07&amp;#34;&lt;/span&gt;     &lt;span class=&#34;c1&#34;&gt;# Set the terminal title to the host we are sshing to&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;sr&#34;&gt;/usr/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;bin&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;/&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;ssh&lt;/span&gt; &lt;span class=&#34;vg&#34;&gt;$1&lt;/span&gt; &lt;span class=&#34;vg&#34;&gt;$2&lt;/span&gt; &lt;span class=&#34;vg&#34;&gt;$3&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;n&#34;&gt;setcolor&lt;/span&gt;  &lt;span class=&#34;c1&#34;&gt;# Once ssh exits, reset the color back to what it should be&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>“My Neckbeard Grew Three Sizes That Day” or How I Beat a GNU tool with Perl</title>
      <link>https://secopsmonkey.com/post/2014-08-01-how-i-beat-a-gnu-tool-with-perl/</link>
      <pubDate>Fri, 01 Aug 2014 00:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-08-01-how-i-beat-a-gnu-tool-with-perl/</guid>
      
        <description>&lt;blockquote&gt;
&lt;p&gt;Pulling back from the archives this is a repost of a previous blog post, with minor edits to include
a more revent version of the app. This time ripped from a guest spot at &lt;a href=&#34;http://thenubbyadmin.com/2012/05/14/my-neckbeard-grew-three-sizes-that-day-or-how-i-beat-a-gnu-tool-with-perl/&#34;&gt;The Nubby Admin&lt;/a&gt;, a
fantastic blog from a fellow tech nerd.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I spend a lot of time doing text based data processing. A &lt;em&gt;lot&lt;/em&gt; of time. During an analysis, I often
want to do things like look at &amp;lsquo;Top Talkers&amp;rsquo;, &amp;lsquo;Most Frequent Visitors&amp;rsquo;, or really anything that
comprises a list of unique identifiers sorted by count. As a result, I&amp;rsquo;ve translated two actions
into a series of pipes:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;What&amp;rsquo;s the count of events per thingy: &lt;code&gt; | sort | uniq -c | sort -n&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Who has been doing whatever: &lt;code&gt; | sort -u&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;This tends to work pretty well in most cases. Today, however, was not one of those cases. While
attempting get a list of unique MACs I started out with a source (i.e. non-uniqued) 16GB text file
with one MAC per line. This is where things got annoying. Muscle memory kicked in and since this
matched Action #2, I ran the following command:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cat macs_all.txt | sort -u &amp;gt; macs_unique.txt
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I expected it to take a few minutes, so I went back to the other things I was doing and let it go. I
checked back 15 minutes later, and it was still running. Waited 5 minutes&amp;hellip;still running. When the
command had been running for 45 minutes, I got fed up and decided that I could do better. Perl,
being my go to tool, came to the rescue in the form of hashes. I won&amp;rsquo;t go into gritty detail, but a
Perl hash is a data structure that consists of a list of key/value pairs. Whenever you assign a
value to a key it will add an entry for the key if it doesn&amp;rsquo;t exist, or update the value if it does.
Since a key cannot be in the same hash multiple times, it makes for a pretty good hack to generate a
unique list. The full source of the app is included at the end, but the basic data flow is:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Read MAC from input&lt;/li&gt;
&lt;li&gt;If MAC exists in list next, else add to list&lt;/li&gt;
&lt;li&gt;If EOF print all MACs&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This worked significantly better for me. The output was not sorted, but that&amp;rsquo;s fine, I didn&amp;rsquo;t need
it sorted, only unique. The timing information looked a lot better too.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;packs@ node1:~&amp;gt; &lt;span class=&#34;nb&#34;&gt;time&lt;/span&gt; cat macs_all.txt &lt;span class=&#34;p&#34;&gt;|&lt;/span&gt; sort -u &amp;gt; macs_unique.txt
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;real    181m12.417s
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;user    176m13.926s
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;sys     1m42.335s
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;packs@ node1:~&amp;gt; &lt;span class=&#34;nb&#34;&gt;time&lt;/span&gt; cat macs_all.txt &lt;span class=&#34;p&#34;&gt;|&lt;/span&gt; ./fast_uniq.pl &amp;gt; macs_fast_uniqed.txt
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;real    8m9.074s
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;user    7m28.176s
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;sys     0m46.271s&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;The times can&amp;rsquo;t really be directly compared, since output from &lt;code&gt;fast_uniq.pl&lt;/code&gt; isn&amp;rsquo;t actually sorted.
Given the pretty substantial difference I think we can reasonably accept the fact that &lt;code&gt;fast_uniq.pl&lt;/code&gt;
is better in this use case. After seeing this, I&amp;rsquo;m tempted to add some functionality so I stop using
both sort and uniq entirely.&lt;/p&gt;
&lt;p&gt;I&amp;rsquo;m interested to hear if anyone else has done something similar or explain to me how much my code
sucks.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-perl&#34; data-lang=&#34;perl&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;ch&#34;&gt;#!/usr/bin/perl -w&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;use&lt;/span&gt; &lt;span class=&#34;nn&#34;&gt;strict&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;use&lt;/span&gt; &lt;span class=&#34;nn&#34;&gt;Getopt::Long&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;use&lt;/span&gt; &lt;span class=&#34;nn&#34;&gt;Pod::Usage&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Setup main variables from script arguments&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;my&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;nv&#34;&gt;$unique&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$sort&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$count&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$quiet&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$DEBUG&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;);&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Grab the options passed on the command line.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;n&#34;&gt;GetOptions&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;s&#34;&gt;&amp;#34;unique|u&amp;#34;&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;\&lt;/span&gt;&lt;span class=&#34;nv&#34;&gt;$unique&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;              &lt;span class=&#34;c1&#34;&gt;# flag&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;s&#34;&gt;&amp;#34;sort|s&amp;#34;&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;\&lt;/span&gt;&lt;span class=&#34;nv&#34;&gt;$sort&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;                  &lt;span class=&#34;c1&#34;&gt;# flag&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;s&#34;&gt;&amp;#34;count|c&amp;#34;&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;\&lt;/span&gt;&lt;span class=&#34;nv&#34;&gt;$count&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;                &lt;span class=&#34;c1&#34;&gt;# flag&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;s&#34;&gt;&amp;#34;quiet|q&amp;#34;&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;\&lt;/span&gt;&lt;span class=&#34;nv&#34;&gt;$quiet&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;                &lt;span class=&#34;c1&#34;&gt;# flag&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;s&#34;&gt;&amp;#34;verbose|v&amp;#34;&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;\&lt;/span&gt;&lt;span class=&#34;nv&#34;&gt;$DEBUG&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt;              &lt;span class=&#34;c1&#34;&gt;# flag&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;s&#34;&gt;&amp;#34;help|?|h&amp;#34;&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;sub&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pod2usage&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;1&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;);&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;},&lt;/span&gt; &lt;span class=&#34;c1&#34;&gt;# flag&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;ow&#34;&gt;or&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;pod2usage&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;s&#34;&gt;&amp;#34;$0: Unrecognized program argument.&amp;#34;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;);&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;if&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;!&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt; &lt;span class=&#34;nb&#34;&gt;defined&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;nv&#34;&gt;$unique&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;ow&#34;&gt;or&lt;/span&gt; &lt;span class=&#34;nb&#34;&gt;defined&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;nv&#34;&gt;$sort&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;ow&#34;&gt;or&lt;/span&gt; &lt;span class=&#34;nb&#34;&gt;defined&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;nv&#34;&gt;$count&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;n&#34;&gt;pod2usage&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;s&#34;&gt;&amp;#34;$0:  Required argument missing.&amp;#34;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;);&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;my&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;%hash&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$key&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;,&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$value&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;);&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;if&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$unique&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;k&#34;&gt;print&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;&amp;#34;Entered unique loop\n&amp;#34;&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;if&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$DEBUG&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;k&#34;&gt;while&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;my&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$mac&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;&amp;lt;&amp;gt;&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;k&#34;&gt;next&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;unless&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$mac&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;nb&#34;&gt;chomp&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$mac&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;nv&#34;&gt;$hash&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;&lt;span class=&#34;nv&#34;&gt;$mac&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;++&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;c1&#34;&gt;# Do all the prints here&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;k&#34;&gt;if&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$sort&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;k&#34;&gt;print&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;&amp;#34;Entered sorted print loop\n&amp;#34;&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;if&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$DEBUG&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;k&#34;&gt;for&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$key&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt; &lt;span class=&#34;nb&#34;&gt;sort&lt;/span&gt; &lt;span class=&#34;nb&#34;&gt;keys&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;%hash&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;k&#34;&gt;if&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$count&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;        &lt;span class=&#34;k&#34;&gt;print&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;&amp;#34;$hash{$key} \t$key\n&amp;#34;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;k&#34;&gt;else&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;        &lt;span class=&#34;k&#34;&gt;print&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;&amp;#34;$key\n&amp;#34;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;k&#34;&gt;else&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;k&#34;&gt;print&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;&amp;#34;Entered unsorted print loop\n&amp;#34;&lt;/span&gt; &lt;span class=&#34;k&#34;&gt;if&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$DEBUG&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;k&#34;&gt;for&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$key&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;nb&#34;&gt;keys&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;%hash&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;k&#34;&gt;if&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$count&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;        &lt;span class=&#34;k&#34;&gt;print&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;&amp;#34;$hash{$key} \t$key\n&amp;#34;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;k&#34;&gt;else&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;        &lt;span class=&#34;k&#34;&gt;print&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;&amp;#34;$key\n&amp;#34;&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;if&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$sort&lt;/span&gt; &lt;span class=&#34;ow&#34;&gt;and&lt;/span&gt; &lt;span class=&#34;ow&#34;&gt;not&lt;/span&gt; &lt;span class=&#34;nv&#34;&gt;$unique&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;c1&#34;&gt;# Join to an array and sort for print&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;  &lt;span class=&#34;k&#34;&gt;print&lt;/span&gt; &lt;span class=&#34;s&#34;&gt;&amp;#34;So you want sorted but uniqued data, eh? Patience my dear, patience.\n&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;__END__
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;=head1 NAME
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;fast_sorter - Reimplementation of GNU sort and GNU uniq to perform faster
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;by Scott Pack
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;=head1 SYNOPSIS
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;fast_sorter.pl [options]
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt; Options:
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;   -u, --unique     Deduplicate entries in the input
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;   -s, --sort       Print the output ASCII sorted
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;   -c, --count      Print the output along with duplication counts
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;   -v, --verbose    Print debugging information
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;   -h, --help       Brief help message
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;By Scott Pack
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;cp&#34;&gt;=cut&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>On Community</title>
      <link>https://secopsmonkey.com/post/2014-07-25-on_community/</link>
      <pubDate>Fri, 25 Jul 2014 00:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-07-25-on_community/</guid>
      
        <description>&lt;p&gt;Some few of us, primarily in consultancy and professional services, are in the position to work in
a place surrounded by other Information Security people but for most of us the ratios are a little
different. To speak from personal experience, at my previous company there were 6 of us out of a
total IT staff of about 170. Based on conversations with others my experience seems somewhat
typical, or maybe even high. Currently the ratio is a little skewed since I am the information
security staff, although we are admittedly a small company. Being part of such a small team means
it can be awfully hard to keep from falling into familiar patterns.&lt;/p&gt;
&lt;p&gt;There&amp;rsquo;s also quite a lot to be said for having a support network. We are often seen as the road
block, the group who can always be relied on to say no. We frequently find ourselves ignored by
management. We spend one day installing software patches and the next investigating some rather
&amp;hellip;let&amp;rsquo;s say disheartening activity. We&amp;rsquo;re a group who are required to always imagine as many of
the worst case possible scenarios and &lt;em&gt;hope&lt;/em&gt; we can come up with a way to avoid them. Hell it&amp;rsquo;s
often said, half-jokingly I&amp;rsquo;d like to hope, that cynicism is a necessity to succeed. At the end of
the day is it any wonder that InfoSec Burnout is enough of a thing that conferences are hosting
panel discussions on the matter and have spawned sites like &lt;a href=&#34;http://www.itburnout.org/&#34;&gt;Information Technology Burnout Project&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;That&amp;rsquo;s not to say we&amp;rsquo;re at all unique in this respect, all careers have their own flavor of stress
points. Randall Monroe does a good job of illustrating the Cross born by our Operations siblings
&lt;a href=&#34;http://xkcd.com/705/&#34;&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-07-25-on_community/devotion_to_duty.png&#34;
    alt=&#34;The weird sense of duty really good sysadmins have can border on the sociopathic, but it&amp;#39;s nice to know that it stands between the forces of darkness and your cat blog&amp;#39;s servers.&#34;&gt;
&lt;/figure&gt;

&lt;p&gt;The difference, I would argue, is that in most situations they are more likely to be able to stalk
out of their office and find a kindred soul to steam at. Given that we&amp;rsquo;re more likely to work alone
all we can do is carry those problems home and hope sleep, or drink, them away. Quite honestly, I
need a better solution.&lt;/p&gt;
&lt;p&gt;This past weekend I attended &lt;a href=&#34;http://bsidescle.com/&#34;&gt;BSidesCleveland&lt;/a&gt;. According to the organizers we had just under 200
people between attendees, speakers, and staff all housed in a fantastically relaxed venue. That&amp;rsquo;s
nearly 200 people who have the same problems, same concerns, and most importantly rarely the same
opinions. Driving home I was left feeling tired but also refreshed and with a couple of new projects
in mind. I learned a few things, adjusted my perspective on others, and spent quality time talking
to a few people that I hope to keep in contact with.&lt;/p&gt;
&lt;p&gt;The real lesson is that events like this are critical for us. They provide a neutral location where
none of us have to feel like we&amp;rsquo;re on display and can treat each other like the kindred spirits we
really are.&lt;/p&gt;
&lt;p&gt;I &lt;em&gt;highly&lt;/em&gt; encourage you to look in your local area for professional events to attend. In the
Columbus area we have &lt;a href=&#34;http://www.eventbrite.com/o/isc2-central-ohio-chapter-smba-318552366&#34;&gt;SecurityMBA&lt;/a&gt; and &lt;a href=&#34;https://www.eventbrite.com/e/cmhseclunch-tickets-5916669919&#34;&gt;CMHSecLunch&lt;/a&gt;, both of which are monthly social events
targeting the Information Security community. I&amp;rsquo;ve also really enjoyed the events put on by our
&lt;a href=&#34;https://lists.lopsa.org/cgi-bin/mailman/listinfo/lopsa-us-oh-columbus/&#34;&gt;League of Professional Systems Administrator chapter&lt;/a&gt;. While not InfoSec specific there is a
good crew of compliance and security related people mixed in with the Operations folks. If you&amp;rsquo;re
not local to me, and will forgive the self-promotion, check out &lt;a href=&#34;http://secopsmonkey.com/infosechappenings/&#34;&gt;InfoSec Happenings&lt;/a&gt; and see if
you can find an event in your area. Don&amp;rsquo;t forget the power of social media. Twitter is rotten with
&lt;em&gt;us people&lt;/em&gt;.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>What&#39;s Up With the HIPAA Retention Schedules?</title>
      <link>https://secopsmonkey.com/post/2014-07-18-whats-up-with-the-hipaa-retention-schedules/</link>
      <pubDate>Sat, 19 Jul 2014 00:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-07-18-whats-up-with-the-hipaa-retention-schedules/</guid>
      
        <description>&lt;p&gt;For much of my professional career I have been what I like to call &amp;ldquo;health care adjacent&amp;rdquo;. Meaning,
my department has HIPAA responsibilities but I personally was only partially involved in them. I
was around for conversations around protecting PHI, e- or otherwise, and sometimes called in as a
technical resource for addressing specific controls but I was never in a situation where I &lt;em&gt;really&lt;/em&gt;
needed to know shit about the standard. One truism that everyone involved clearly knew was that
records retention was a nightmare. We had a hard requirement of a 7-year minimum for anything
related to patient data, whether the patient&amp;rsquo;s electronic chart or firewall logs for the
network segment that housed HIPAA covered servers.&lt;/p&gt;
&lt;p&gt;Working for a company that services hospitals and medical services providers has meant stepping
away from my former blissful ignorance and digging down into the absolute mess that is HIPAA and
all of its descendants. What struck me is that I never quite saw the declaration of that 7-year
requirement. I simply assumed I hadn&amp;rsquo;t quite gotten to that section, or that it was listed in one
of the follow-up bills. After all, everyone I&amp;rsquo;ve spoken to has treated the 7-year requirement as
fact, both at my previous employer and others. Needless to say, I was rather surprised, when I
learned that there&amp;rsquo;s not actually any such requirement at all!&lt;/p&gt;
&lt;p&gt;In fact, the only legislated retention schedule at all is found in 45 CFR 164.316(b)(2)(i).&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Time limit (Required). Retain the documentation required by paragraph (b)(1) of this section for
6 years from the date of its creation or the date when it last was in effect, whichever is later.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Yes, 6 years is awfully close to 7 but the given section entirely revolves around policy and
procedure. I think it should be pointed out that policies and procedure are slightly different from
health records, and very different from transaction artifacts such as firewall logs.&lt;/p&gt;
&lt;p&gt;That&amp;rsquo;s not to say we don&amp;rsquo;t have &lt;em&gt;any&lt;/em&gt; retention schedules to follow, they&amp;rsquo;re just not defined by
HIPAA et al. Take the Centers for Medicare &amp;amp; Medicaid Services (CMS) for example. They define
several schedules dependent on the type of information; provider submitted cost reports must
be retained in their original or legally reproduced form for a period of at least 5 years after
the closure of the cost report (42 CFR 482.24(b)(1))&lt;a href=&#34;http://www.access.gpo.gov/nara/cfr/waisidx_05/42cfr482_05.html&#34;&gt;1&lt;/a&gt;, or Medicare managed care program providers
to retain records for 10 years (42 CFR 422.504(d)(2)(iii))&lt;a href=&#34;http://www.law.cornell.edu/cfr/text/42/422.504&#34;&gt;2&lt;/a&gt;, or psychiatric hospitals for 5
years (42 CFR 482.61)&lt;a href=&#34;http://www.law.cornell.edu/cfr/text/42/482.61&#34;&gt;3&lt;/a&gt;. The American Health Information Management Association has done a
pretty fantastic job at summarizing many, though almost definitely not all, of the schedules
in &amp;lsquo;Practice Brief—Retention of Health Information&amp;rsquo;&lt;a href=&#34;http://library.ahima.org/xpedio/groups/public/documents/ahima/bok1_012546.pdf&#34;&gt;4&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Looking through the list we can see an amazingly large number of individual retention schedules
with time frames that are all over the place. While that is a great resource it doesn&amp;rsquo;t really
help understand the whole 7-Year Mystery that prompted my investigation. The closest thing I&amp;rsquo;ve
found to an authoritative source is from the State of Connecticut Public Health Code.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;19a-14-42. Retention schedule
Unless specified otherwise herein, all parts of a medical record shall be retained for a period of
seven (7) years from the last date of treatment, or, upon the death of the patient, for three (3)
years.
(a) Pathology Slides, EEG and ECG Tracings must each be kept for seven (7) years. If an
ECG is taken and the results are unchanged from a previous ECG, then only the most
recent results need be retained. Reports on each of these must e kept for the duration of
the medical record.
(b) Lab Reports and PKU Reports must be kept for at least five (5) years. Only positive
(abnormal) lab results need be retained.
(c) X-Ray Films must be kept for three (3) years.
(Effective August 29, 1984.)&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Per the HHS all of the requirements spelled out in HIPAA et al were intended to be minimums. That
is, if your state has overlapping guidelines you must follow the stricter of the two. My best guess
is that at one time an audit performed in Connecticut and listed 7-year retention as a finding.
Audit findings being what they are found their way into the public consciousness and became so
deeply embedded in compliance lore that most practitioners were unwilling to question it or
suffer the disappointment that is reading US federal legislation to discover what really is
required.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>Using Office 365 as a Smart Host with Postfix</title>
      <link>https://secopsmonkey.com/post/2014-07-11-mail-relaying-postfix-through-office-365/</link>
      <pubDate>Fri, 11 Jul 2014 00:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-07-11-mail-relaying-postfix-through-office-365/</guid>
      
        <description>&lt;blockquote&gt;
&lt;p&gt;Since writing this post I&amp;rsquo;ve learned a better way. If you&amp;rsquo;re using a personal
account or only need to relay 1 server the below may be sufficient. If you&amp;rsquo;re
managing more than one server and can manage your Office365 domain please see
my updated post &lt;a href=&#34;http://secopsmonkey.com/better-mail-relaying-postfix-through-office-365.html&#34;&gt;&lt;em&gt;Better Use of Office 365 as a Smart Host with Postfix&lt;/em&gt;&lt;/a&gt;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;One great thing about Linux systems is that we can get automagic reporting on
so many things just by setting up a mail server. Want to run a command and find
out if it failed? Throw it in &lt;code&gt;cron&lt;/code&gt; and redirect the output to &lt;code&gt;stdout&lt;/code&gt;. Want
to write a script to generate a report? Save it as ASCII text and run the &lt;code&gt;mail&lt;/code&gt;
command. Bam! Instant report distribution. Also because of how easy it is to
interact with the mailing subsystem many applications are written with it in
mind. Just like how logging is, largely, standardized on syslog reporting has
largely standardized on email.&lt;/p&gt;
&lt;p&gt;In the Sendmail world we could set this up fairly easily using a SmartHost. Even
authenticated is not that big a deal, though truthfully that&amp;rsquo;s mostly because I
worked out a successful config aeons ago that is easily modified. These days,
however, I&amp;rsquo;m working in an Ubuntu environment which now has a pretty strong
preference for postfix. Fortunately I was a somewhat easy grognard to retrain,
the syntax is certainly easier. Just as with sendmail you start with setting
your SmartHost, now called a relayhost&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;relayhost = [smtp.office365.com]:587
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Next up we&amp;rsquo;ll need to define the user credentials necessary to send mail through
our relayhost. Just as with sendmail we&amp;rsquo;ll create a file containing the
credentials, say &lt;code&gt;/etc/postfix/sasl_passwd&lt;/code&gt;, with the contents of&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;[smtp.office365.com]:587 user@domain.tld:soopersekretPassvv0rd
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;In order to turn this into a file that postfix can use we&amp;rsquo;ll run it through a
processor.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;postmap hash:/etc/postfix/sasl_passwd
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This generates a BerkDB file based on the raw text file we created before. Now
we tell postfix to use it by adding this to the &lt;code&gt;main.cf&lt;/code&gt;.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;In theory this should be sufficient but it&amp;rsquo;s actually not. If you&amp;rsquo;ll notice I&amp;rsquo;m
relaying through Office365. Their mailers have an additional restriction, first
you must authenticate using a valid username and password then your user must
be allowed to send messages as the email address given in the From header.&lt;/p&gt;
&lt;p&gt;Mail coming from Linux services can be all over the map. Typically &lt;code&gt;cron&lt;/code&gt; output
will be &lt;code&gt;root@servername&lt;/code&gt;, webapps may send as &lt;code&gt;no-reply@servername.tld&lt;/code&gt;, others
like logwatch, may use &lt;code&gt;application@servername&lt;/code&gt;. All of those will be rejected
by the relayhost since our service account user doesn&amp;rsquo;t have those permissions.
Postfix lets us do this using what they call a &lt;code&gt;generic-map&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Create the map file, here &lt;code&gt;/etc/postfix/generic&lt;/code&gt; with&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;/.*/    user@domain.tld
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then just as above we&amp;rsquo;ll turn it into a database&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;postmap hash:/etc/postfix/generic
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;And update &lt;code&gt;main.cf&lt;/code&gt; to look for the file.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;smtp_generic_maps = hash:/etc/postfix/generic
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now &lt;strong&gt;all&lt;/strong&gt; outgoing messages will have the From field replaced in both the
envelope and header. The first field, the &lt;code&gt;/.*/&lt;/code&gt;, is a regular expression that
defines everything. You can replace that with more specific addresses, more
details are in the &lt;a href=&#34;http://www.postfix.org/ADDRESS_REWRITING_README.html#generic&#34;&gt;official docs&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Now you can just restart the postfix service and start routing your mail. Well&amp;hellip;
nearly. Office365 has some additional settings so I&amp;rsquo;ll include a full config at
the bottom.&lt;/p&gt;
&lt;p&gt;All we&amp;rsquo;ve done so far is just set up relaying for &lt;em&gt;outgoing&lt;/em&gt; messages. Almost
all applications that prompted this work send their messages to &lt;code&gt;root&lt;/code&gt;. This
isn&amp;rsquo;t really all that grand since in order to read the messages you&amp;rsquo;ll have to
log into the host directly. What we &lt;em&gt;really&lt;/em&gt; want to do is have these go
someplace else. Edit the file &lt;code&gt;/etc/aliases&lt;/code&gt; and look for a line where the first
entry is &lt;code&gt;root&lt;/code&gt;. Make it look like this.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;root: scottpack+totesspam@hotmale.com
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Next run the command &lt;code&gt;/usr/bin/newaliases&lt;/code&gt; and all messages sent to &lt;code&gt;root&lt;/code&gt; will
get forwarded to my system reports address. If the applications you want to get
messages from send do a different address, for example &lt;code&gt;apt-mirror&lt;/code&gt; really likes
sending mail to itself, then add additional lines for each local user name. Just
make sure to run &lt;code&gt;newaliases&lt;/code&gt; each time you save the file.&lt;/p&gt;
&lt;p&gt;Now, with the rest of the changes to your &lt;code&gt;main.cf&lt;/code&gt; listed below, you should
start getting all those great email alerts all routed through your spiffy O365
server.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;relayhost = [smtp.office365.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_generic_maps = hash:/etc/postfix/generic
smtp_tls_security_level = may
smtp_sasl_security_options = noanonymous
&lt;/code&gt;&lt;/pre&gt;
</description>
      
    </item>
    
    <item>
      <title>SSL Chain Cert Fun with Nessus</title>
      <link>https://secopsmonkey.com/post/2014-07-04-ssl-chain-cert-fun-with-nessus/</link>
      <pubDate>Fri, 04 Jul 2014 00:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-07-04-ssl-chain-cert-fun-with-nessus/</guid>
      
        <description>&lt;blockquote&gt;
&lt;p&gt;Pulling back from the archives this is a repost of a previous blog post. This time ripped from a
guest spot at &lt;a href=&#34;http://security.blogoverflow.com/2012/04/ssl-chain-cert-fun-with-nessus/&#34;&gt;The Security Stack Exchange Community Blog&lt;/a&gt;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;My workplace recently, for some definitions of recent, switched the company we use for certificate
signing to InCommon. There were quite a few technical/administrative advantages, and since we&amp;rsquo;re
educational, price was a big factor. Everyone has been really happy with the results. Well, except
for this one thing. InCommon is not a top level trusted CA, they chain through AddTrust. This isn&amp;rsquo;t
actually all that big a deal, really, as AddTrust is a common CA to have in your trusted bundle, and
all we had to do was configure the InCommon chain certificate on our web servers. Other than the
occasional chain breakage on some mobile browsers everything seemed peachy. Except, that is, when
we ran a vulnerability scan.&lt;/p&gt;
&lt;p&gt;Shortly after we switched we started noticing some odd alerts coming out of our vulnerability scans.
At first one or two were reporting that the SSL certificate could not be validated. We manually
verified the certificates, declared them as false positives, and moved on. Over time more and more
systems started reporting this error. Eventually the problem had propagated out far enough that I
started digging into it. For reference, the PluginID we&amp;rsquo;re looking at here is &lt;a href=&#34;http://www.nessus.org/plugins/index.php?view=single&amp;amp;id=51192&#34;&gt;51192&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;I learned two very important, and relevant, pieces of information that day:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Nessus was not properly validating the chain.&lt;/li&gt;
&lt;li&gt;Chain Certificate files are a little stranger than expected.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Instead of using a system default CA bundle, Nessus ships with its own. You can find the bundle,
called &lt;code&gt;known_CA.inc&lt;/code&gt;, in the plugin directory. So on Linux systems you should be looking at
&lt;code&gt;/opt/nessus/lib/nessus/plugins/known_CA.inc&lt;/code&gt;. If you are using a Windows scanner, well, you&amp;rsquo;re on
your own. This is a fairly standard looking CA bundle, and I found that AddTrust was, in fact,
included. I did not, however, find any reference to InCommon. Since they are somewhat related to
Internet2 I looked for them, also no luck.&lt;/p&gt;
&lt;p&gt;This isn&amp;rsquo;t really that big a deal, though. Nessus will also look for, but will not update, a
secondary bundle called &lt;code&gt;custom_CA.inc&lt;/code&gt;. In most cases, this file would be used to include a local
CA, for instance in a closed corporate network where one generates self-signed certificates as
a matter of course. However, since you can use it to include arbitrary CA certs we can use it to
fix our problem.&lt;/p&gt;
&lt;p&gt;It&amp;rsquo;s easy enough for me to get the intermediate cert, what with it being public and all. This is
where things started to get a little weird, though. In order to stay consistent with the
&lt;code&gt;known_CA.inc&lt;/code&gt; I included the certificate as a decoded X.509+PEM. Placing only the intermediate cert
in this file resulted in, again, the certificate chain failing to validate. Next, what follows is a
Nessus debugging tip that was roughly an hour&amp;rsquo;s worth of swearing in the discovering:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;If you don&amp;rsquo;t think the web interface is showing you sufficient information, look at the plugin output in the raw XML.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;You can get this by either exporting the report, or by finding it in the user&amp;rsquo;s reports folder on
the scanner. What I discovered was that all of the various and sundry certificates were being read
and validated. The chain, however, was being checked in the wrong order, in this case:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;webserver -&amp;gt; AddTrust -&amp;gt; InCommon.
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;After a little more trial and error I learned that, not only, did I need to have both the InCommon
intermediate, but also the AddTrust certificates in my &lt;code&gt;custom_CA.inc&lt;/code&gt; file, but that the order of
the certs in the file also mattered. As it happens, AddTrust had to be entered first, followed by
InCommon. This does make some amount of sense, when I adjusted my thought process to an actual chain
where AddTrust was the &amp;ldquo;top-level&amp;rdquo;.&lt;/p&gt;
&lt;p&gt;For completeness, I copied the newly complete &lt;code&gt;custom_CA.inc&lt;/code&gt; file to my test webserver and included
it as a chain cert using the &lt;code&gt;SSLCertificateChainFile&lt;/code&gt; option. This is Apache httpd on Linux, you
nginx or IIS folks are on your own. After removing the &lt;code&gt;custom_CA.inc&lt;/code&gt; on the Nessus scanner and
re-running the scan resulted in the certificate properly validating.&lt;/p&gt;
&lt;p&gt;This left me in a good place in two ways:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;I now had a properly formatted &lt;code&gt;custom_CA.inc&lt;/code&gt; file that I could put into puppet for all the scanners.&lt;/li&gt;
&lt;li&gt;I now also had a properly formatted chain cert file for inclusion on the web servers.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;This fixes the problem from both sides, the server presenting all the correct information, as well
as the scanner for cleaning up a false positive. For reference, included below is the chain cert
file that was generated. As mentioned previously, it is the same format as a CA bundle. For each
certificate you’ll find the ASCII text decoded certificate information, followed by the Base64
encoded PEM version of the same certificate. In my testing, Nessus would accept only the PEM
versions, however I wanted to include both outputs since it appears to be the standard.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            7f:71:c1:d3:a2:26:b0:d2:b1:13:f3:e6:81:67:64:3e
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=SE, O=AddTrust AB, OU=AddTrust External TTP Network, CN=AddTrust External CA Root
        Validity
            Not Before: Dec  7 00:00:00 2010 GMT
            Not After : May 30 10:48:38 2020 GMT
        Subject: C=US, O=Internet2, OU=InCommon, CN=InCommon Server CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:97:7c:c7:c8:fe:b3:e9:20:6a:a3:a4:4f:8e:8e:
                    34:56:06:b3:7a:6c:aa:10:9b:48:61:2b:36:90:69:
                    e3:34:0a:47:a7:bb:7b:de:aa:6a:fb:eb:82:95:8f:
                    ca:1d:7f:af:75:a6:a8:4c:da:20:67:61:1a:0d:86:
                    c1:ca:c1:87:af:ac:4e:e4:de:62:1b:2f:9d:b1:98:
                    af:c6:01:fb:17:70:db:ac:14:59:ec:6f:3f:33:7f:
                    a6:98:0b:e4:e2:38:af:f5:7f:85:6d:0e:74:04:9d:
                    f6:27:86:c7:9b:8f:e7:71:2a:08:f4:03:02:40:63:
                    24:7d:40:57:8f:54:e0:54:7e:b6:13:48:61:f1:de:
                    ce:0e:bd:b6:fa:4d:98:b2:d9:0d:8d:79:a6:e0:aa:
                    cd:0c:91:9a:a5:df:ab:73:bb:ca:14:78:5c:47:29:
                    a1:ca:c5:ba:9f:c7:da:60:f7:ff:e7:7f:f2:d9:da:
                    a1:2d:0f:49:16:a7:d3:00:92:cf:8a:47:d9:4d:f8:
                    d5:95:66:d3:74:f9:80:63:00:4f:4c:84:16:1f:b3:
                    f5:24:1f:a1:4e:de:e8:95:d6:b2:0b:09:8b:2c:6b:
                    c7:5c:2f:8c:63:c9:99:cb:52:b1:62:7b:73:01:62:
                    7f:63:6c:d8:68:a0:ee:6a:a8:8d:1f:29:f3:d0:18:
                    ac:ad
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A


            X509v3 Subject Key Identifier:
                48:4F:5A:FA:2F:4A:9A:5E:E0:50:F3:6B:7B:55:A5:DE:F5:BE:34:5D
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Certificate Policies:
                Policy: X509v3 Any Policy



            X509v3 CRL Distribution Points:



                Full Name:
                  URI:http://crl.usertrust.com/AddTrustExternalCARoot.crl



            Authority Information Access:
                CA Issuers - URI:http://crt.usertrust.com/AddTrustExternalCARoot.p7c
                CA Issuers - URI:http://crt.usertrust.com/AddTrustUTNSGCCA.crt
                OCSP - URI:http://ocsp.usertrust.com



    Signature Algorithm: sha1WithRSAEncryption
        93:66:21:80:74:45:85:4b:c2:ab:ce:32:b0:29:fe:dd:df:d6:
        24:5b:bf:03:6a:6f:50:3e:0e:1b:b3:0d:88:a3:5b:ee:c4:a4:
        12:3b:56:ef:06:7f:cf:7f:21:95:56:3b:41:31:fe:e1:aa:93:
        d2:95:f3:95:0d:3c:47:ab:ca:5c:26:ad:3e:f1:f9:8c:34:6e:
        11:be:f4:67:e3:02:49:f9:a6:7c:7b:64:25:dd:17:46:f2:50:
        e3:e3:0a:21:3a:49:24:cd:c6:84:65:68:67:68:b0:45:2d:47:
        99:cd:9c:ab:86:29:11:72:dc:d6:9c:36:43:74:f3:d4:97:9e:
        56:a0:fe:5f:40:58:d2:d5:d7:7e:7c:c5:8e:1a:b2:04:5c:92:
        66:0e:85:ad:2e:06:ce:c8:a3:d8:eb:14:27:91:de:cf:17:30:
        81:53:b6:66:12:ad:37:e4:f5:ef:96:5c:20:0e:36:e9:ac:62:
        7d:19:81:8a:f5:90:61:a6:49:ab:ce:3c:df:e6:ca:64:ee:82:
        65:39:45:95:16:ba:41:06:00:98:ba:0c:56:61:e4:c6:c6:86:
        01:cf:66:a9:22:29:02:d6:3d:cf:c4:2a:8d:99:de:fb:09:14:
        9e:0e:d1:d5:c6:d7:81:dd:ad:24:ab:ac:07:05:e2:1d:68:c3:
        70:66:5f:d3
-----BEGIN CERTIFICATE-----
MIIEwzCCA6ugAwIBAgIQf3HB06ImsNKxE/PmgWdkPjANBgkqhkiG9w0BAQUFADBv
MQswCQYDVQQGEwJTRTEUMBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFk
ZFRydXN0IEV4dGVybmFsIFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBF
eHRlcm5hbCBDQSBSb290MB4XDTEwMTIwNzAwMDAwMFoXDTIwMDUzMDEwNDgzOFow
UTELMAkGA1UEBhMCVVMxEjAQBgNVBAoTCUludGVybmV0MjERMA8GA1UECxMISW5D
b21tb24xGzAZBgNVBAMTEkluQ29tbW9uIFNlcnZlciBDQTCCASIwDQYJKoZIhvcN
AQEBBQADggEPADCCAQoCggEBAJd8x8j+s+kgaqOkT46ONFYGs3psqhCbSGErNpBp
4zQKR6e7e96qavvrgpWPyh1/r3WmqEzaIGdhGg2GwcrBh6+sTuTeYhsvnbGYr8YB
+xdw26wUWexvPzN/ppgL5OI4r/V/hW0OdASd9ieGx5uP53EqCPQDAkBjJH1AV49U
4FR+thNIYfHezg69tvpNmLLZDY15puCqzQyRmqXfq3O7yhR4XEcpocrFup/H2mD3
/+d/8tnaoS0PSRan0wCSz4pH2U341ZVm03T5gGMAT0yEFh+z9SQfoU7e6JXWsgsJ
iyxrx1wvjGPJmctSsWJ7cwFif2Ns2Gig7mqojR8p89AYrK0CAwEAAaOCAXcwggFz
MB8GA1UdIwQYMBaAFK29mHo0tCb3+sQmVO8DveAky1QaMB0GA1UdDgQWBBRIT1r6
L0qaXuBQ82t7VaXe9b40XTAOBgNVHQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB
/wIBADARBgNVHSAECjAIMAYGBFUdIAAwRAYDVR0fBD0wOzA5oDegNYYzaHR0cDov
L2NybC51c2VydHJ1c3QuY29tL0FkZFRydXN0RXh0ZXJuYWxDQVJvb3QuY3JsMIGz
BggrBgEFBQcBAQSBpjCBozA/BggrBgEFBQcwAoYzaHR0cDovL2NydC51c2VydHJ1
c3QuY29tL0FkZFRydXN0RXh0ZXJuYWxDQVJvb3QucDdjMDkGCCsGAQUFBzAChi1o
dHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vQWRkVHJ1c3RVVE5TR0NDQS5jcnQwJQYI
KwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnVzZXJ0cnVzdC5jb20wDQYJKoZIhvcNAQEF
BQADggEBAJNmIYB0RYVLwqvOMrAp/t3f1iRbvwNqb1A+DhuzDYijW+7EpBI7Vu8G
f89/IZVWO0Ex/uGqk9KV85UNPEerylwmrT7x+Yw0bhG+9GfjAkn5pnx7ZCXdF0by
UOPjCiE6SSTNxoRlaGdosEUtR5nNnKuGKRFy3NacNkN089SXnlag/l9AWNLV1358
xY4asgRckmYOha0uBs7Io9jrFCeR3s8XMIFTtmYSrTfk9e+WXCAONumsYn0ZgYr1
kGGmSavOPN/mymTugmU5RZUWukEGAJi6DFZh5MbGhgHPZqkiKQLWPc/EKo2Z3vsJ
FJ4O0dXG14HdrSSrrAcF4h1ow3BmX9M=
-----END CERTIFICATE-----
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=SE, O=AddTrust AB, OU=AddTrust External TTP Network, CN=AddTrust External CA Root
        Validity
            Not Before: May 30 10:48:38 2000 GMT
            Not After : May 30 10:48:38 2020 GMT
        Subject: C=SE, O=AddTrust AB, OU=AddTrust External TTP Network, CN=AddTrust External CA Root
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:b7:f7:1a:33:e6:f2:00:04:2d:39:e0:4e:5b:ed:
                    1f:bc:6c:0f:cd:b5:fa:23:b6:ce:de:9b:11:33:97:
                    a4:29:4c:7d:93:9f:bd:4a:bc:93:ed:03:1a:e3:8f:
                    cf:e5:6d:50:5a:d6:97:29:94:5a:80:b0:49:7a:db:
                    2e:95:fd:b8:ca:bf:37:38:2d:1e:3e:91:41:ad:70:
                    56:c7:f0:4f:3f:e8:32:9e:74:ca:c8:90:54:e9:c6:
                    5f:0f:78:9d:9a:40:3c:0e:ac:61:aa:5e:14:8f:9e:
                    87:a1:6a:50:dc:d7:9a:4e:af:05:b3:a6:71:94:9c:
                    71:b3:50:60:0a:c7:13:9d:38:07:86:02:a8:e9:a8:
                    69:26:18:90:ab:4c:b0:4f:23:ab:3a:4f:84:d8:df:
                    ce:9f:e1:69:6f:bb:d7:42:d7:6b:44:e4:c7:ad:ee:
                    6d:41:5f:72:5a:71:08:37:b3:79:65:a4:59:a0:94:
                    37:f7:00:2f:0d:c2:92:72:da:d0:38:72:db:14:a8:
                    45:c4:5d:2a:7d:b7:b4:d6:c4:ee:ac:cd:13:44:b7:
                    c9:2b:dd:43:00:25:fa:61:b9:69:6a:58:23:11:b7:
                    a7:33:8f:56:75:59:f5:cd:29:d7:46:b7:0a:2b:65:
                    b6:d3:42:6f:15:b2:b8:7b:fb:ef:e9:5d:53:d5:34:
                    5a:27
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A
            X509v3 Key Usage:
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Authority Key Identifier:
                keyid:AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A
                DirName:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
                serial:01



    Signature Algorithm: sha1WithRSAEncryption
        b0:9b:e0:85:25:c2:d6:23:e2:0f:96:06:92:9d:41:98:9c:d9:
        84:79:81:d9:1e:5b:14:07:23:36:65:8f:b0:d8:77:bb:ac:41:
        6c:47:60:83:51:b0:f9:32:3d:e7:fc:f6:26:13:c7:80:16:a5:
        bf:5a:fc:87:cf:78:79:89:21:9a:e2:4c:07:0a:86:35:bc:f2:
        de:51:c4:d2:96:b7:dc:7e:4e:ee:70:fd:1c:39:eb:0c:02:51:
        14:2d:8e:bd:16:e0:c1:df:46:75:e7:24:ad:ec:f4:42:b4:85:
        93:70:10:67:ba:9d:06:35:4a:18:d3:2b:7a:cc:51:42:a1:7a:
        63:d1:e6:bb:a1:c5:2b:c2:36:be:13:0d:e6:bd:63:7e:79:7b:
        a7:09:0d:40:ab:6a:dd:8f:8a:c3:f6:f6:8c:1a:42:05:51:d4:
        45:f5:9f:a7:62:21:68:15:20:43:3c:99:e7:7c:bd:24:d8:a9:
        91:17:73:88:3f:56:1b:31:38:18:b4:71:0f:9a:cd:c8:0e:9e:
        8e:2e:1b:e1:8c:98:83:cb:1f:31:f1:44:4c:c6:04:73:49:76:
        60:0f:c7:f8:bd:17:80:6b:2e:e9:cc:4c:0e:5a:9a:79:0f:20:
        0a:2e:d5:9e:63:26:1e:55:92:94:d8:82:17:5a:7b:d0:bc:c7:
        8f:4e:86:04
-----BEGIN CERTIFICATE-----
MIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU
MBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFs
IFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290
MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowbzELMAkGA1UEBhMCU0Ux
FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h
bCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9v
dDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALf3GjPm8gAELTngTlvt
H7xsD821+iO2zt6bETOXpClMfZOfvUq8k+0DGuOPz+VtUFrWlymUWoCwSXrbLpX9
uMq/NzgtHj6RQa1wVsfwTz/oMp50ysiQVOnGXw94nZpAPA6sYapeFI+eh6FqUNzX
mk6vBbOmcZSccbNQYArHE504B4YCqOmoaSYYkKtMsE8jqzpPhNjfzp/haW+710LX
a0Tkx63ubUFfclpxCDezeWWkWaCUN/cALw3CknLa0Dhy2xSoRcRdKn23tNbE7qzN
E0S3ySvdQwAl+mG5aWpYIxG3pzOPVnVZ9c0p10a3CitlttNCbxWyuHv77+ldU9U0
WicCAwEAAaOB3DCB2TAdBgNVHQ4EFgQUrb2YejS0Jvf6xCZU7wO94CTLVBowCwYD
VR0PBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wgZkGA1UdIwSBkTCBjoAUrb2YejS0
Jvf6xCZU7wO94CTLVBqhc6RxMG8xCzAJBgNVBAYTAlNFMRQwEgYDVQQKEwtBZGRU
cnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0ZXJuYWwgVFRQIE5ldHdvcmsx
IjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENBIFJvb3SCAQEwDQYJKoZIhvcN
AQEFBQADggEBALCb4IUlwtYj4g+WBpKdQZic2YR5gdkeWxQHIzZlj7DYd7usQWxH
YINRsPkyPef89iYTx4AWpb9a/IfPeHmJIZriTAcKhjW88t5RxNKWt9x+Tu5w/Rw5
6wwCURQtjr0W4MHfRnXnJK3s9EK0hZNwEGe6nQY1ShjTK3rMUUKhemPR5ruhxSvC
Nr4TDea9Y355e6cJDUCrat2PisP29owaQgVR1EX1n6diIWgVIEM8med8vSTYqZEX
c4g/VhsxOBi0cQ+azcgOno4uG+GMmIPLHzHxREzGBHNJdmAPx/i9F4BrLunMTA5a
mnkPIAou1Z5jJh5VkpTYghdae9C8x49OhgQ=
-----END CERTIFICATE-----
&lt;/code&gt;&lt;/pre&gt;
</description>
      
    </item>
    
    <item>
      <title>My Not So Fancy .screenrc</title>
      <link>https://secopsmonkey.com/post/2014-06-24-my-not-so-fancy-screenrc/</link>
      <pubDate>Tue, 24 Jun 2014 12:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-06-24-my-not-so-fancy-screenrc/</guid>
      
        <description>&lt;p&gt;Some number of days ago my dear &lt;a href=&#34;https://twitter.com/Nonapeptide&#34;&gt;Wesley&lt;/a&gt; put out the call absolutely &lt;em&gt;begging&lt;/em&gt; for our &lt;a href=&#34;http://thenubbyadmin.com/2014/06/23/your-best-screenrc-files-are-wanted/&#34;&gt;Best .screenrc
Files&lt;/a&gt;. While mine is very simple it hits all my special points. First the config, then the
explanation.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;startup_message off
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;chdir
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;defutf8 on
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;utf8 on on
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Enable window monitoring &amp;amp; notifications&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;monitor on
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;defmonitor on
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;c1&#34;&gt;# Change escape command from ctrl-a to ctrl-z&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;escape ^za
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;autodetach on
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;hardstatus on
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;hardstatus alwayslastline
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;hardstatus string &lt;span class=&#34;s1&#34;&gt;&amp;#39;%{= kG}[ %{G}%H %{g}][%= %{=kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B}%Y-%m-%d%{W}%{g}]&amp;#39;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;##Go Home!
Normally the &lt;code&gt;chdir&lt;/code&gt; is used on a special case to switch to a new directory, however here I&amp;rsquo;m using
it to force my screen to start in my home directory. This is mostly because I&amp;rsquo;m lazy and don&amp;rsquo;t want
to have to change directories prior to launch.&lt;/p&gt;
&lt;p&gt;##Unicode All The Things
I honestly don&amp;rsquo;t have a fantastic and compelling reason to use UTF-8. This is 2014 and it seems like
it&amp;rsquo;s time to start using Unicode instead of some silly old school character sets. These lines tell
screen to use UTF-8 by default and to go ahead and turn it on. This way if I open a new screen
window it&amp;rsquo;ll have a consistent character set.&lt;/p&gt;
&lt;p&gt;##Ohai Guise! Look Over Here!
When you have multiple screen windows sometimes it can be tricksome to keep track of them all. When
I first started using multiple windows it was because of IRC. I was using &lt;code&gt;irssi&lt;/code&gt;, because I&amp;rsquo;m a
nerd, and freenode because it&amp;rsquo;s IRC. Then I joined an industry group that had a private IRC server
and it just became easier to run two instances of &lt;code&gt;irssi&lt;/code&gt;. This meant two screen windows so
MONITORING! When I get activity in the window this will throw an alert so that I know somebody said
something.&lt;/p&gt;
&lt;p&gt;##Ephemera
By default screen uses ctrl+a in order to break into command mode. Since I also use bash this is
exceptionally annoying. I&amp;rsquo;m changing the escape character to ctrl+z so that I can finally start
using the &amp;lsquo;start of line&amp;rsquo; key combo again.&lt;/p&gt;
&lt;p&gt;The hardstatus line is&amp;hellip;thick. This is the real magic that makes multiple windows worth using.
This sets up a status line on the bottom that lists all the windows, which one is currently active,
which ones have unread activity, their name, and the current date. If you use this status line the
potentially least obvious things are the three most likely statuses you&amp;rsquo;ll see on the windows. They
will be shown immediately after the line number.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;-&lt;/code&gt; : The previously opened window.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;*&lt;/code&gt; : The current active window.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;@&lt;/code&gt; : This window has unseen activity.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;All of this comes together in something that looks like this.&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-06-24-my-not-so-fancy-screenrc/pack_screenrc.png&#34;
    alt=&#34;My Not So Fancy screenrc&#34;&gt;
&lt;/figure&gt;

</description>
      
    </item>
    
    <item>
      <title>Misadventures with tcpdump Filters</title>
      <link>https://secopsmonkey.com/post/2014-06-20-misadventures-with-tcpdump-filters/</link>
      <pubDate>Fri, 20 Jun 2014 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-06-20-misadventures-with-tcpdump-filters/</guid>
      
        <description>&lt;blockquote&gt;
&lt;p&gt;Pulling back from the archives this is a repost of a previous blog post. This time ripped from a
guest spot at &lt;a href=&#34;http://security.blogoverflow.com/2012/08/misadventures-with-tcpdump-filters/&#34;&gt;The Security Stack Exchange Community Blog&lt;/a&gt;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;For quite some time I&amp;rsquo;ve been running into a tricksome situation with tcpdump. While doing analysis
I kept running into the situation where none of my filters would work right. For example, let&amp;rsquo;s
presume I have an existing capture file that was taken off a mirrored port. According to the manpage
for pcap-filter this command is a syntactically valid construction:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;tcpdump -nnr capturefile.pcap host 10.10.15.15
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;It does not, however, produce any output. I can verify that traffic exists for that host by doing:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;tcpdump -nnr capturefile.pcap | grep 10.10.15.15
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This does in fact produce the results I want, but is a pretty unfortunate work-around. Part of what
makes a tool like tcpdump so useful is the highly complex filtering language available.&lt;/p&gt;
&lt;p&gt;I finally sucked up my ego and asked some of my fellows in the security.stackexchange chat room,
&lt;a href=&#34;http://chat.stackexchange.com/rooms/151/conversation/tcpdump-filtering&#34;&gt;The DMZ&lt;/a&gt;. While our conversation wasn&amp;rsquo;t strictly helpful, since they seemed just as puzzled as
me, talking out the problem did help me come up with some better google search terms.&lt;/p&gt;
&lt;p&gt;I discovered the problem is entirely to do with 802.1q tagged packets. Since this pcap was taken
from a mirrored port of a switch using VLANs it follows all the same rules as a trunked interface.
So what that means is that my above filter gets translated as, &amp;ldquo;Look in the source and destination
address fields of the IP header of this standard packet.&amp;rdquo; Anyone who has had to decode packets, or
parse our network traffic, should probably assume that while parsing can be tricky, this lookup
shouldn&amp;rsquo;t be very difficult. I definitely fell into the same boat and boy was I wrong.&lt;/p&gt;
&lt;p&gt;My first assumption was that when applying a BPF to a packet capture the following order of events
occurred:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Read in packet&lt;/li&gt;
&lt;li&gt;Parse packet into identifiable tokens&lt;/li&gt;
&lt;li&gt;Check filter strings against tokenized packet&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;As it turns out, this isn&amp;rsquo;t what happens at all. Our simple filter above is really just a macro. The
host macro will parse the packet, but it is a rather simple parser. By and large this is good, since
we want the filters to be fast. In some situations this is bad. For the purposes of discussion let&amp;rsquo;s
make two assumptions:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;The host macro is nothing more than &amp;ldquo;src host ip or dst host ip&amp;rdquo;&lt;/li&gt;
&lt;li&gt;The src and dst macros are nothing more than &amp;ldquo;src = 13th-16th bytes of IP header&amp;rdquo; and &amp;ldquo;dst = 17th-20th bytes of IP header&amp;rdquo;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;While the macro language as a whole is really much more complicated, this simplistic view is good
enough for this discussion, and in my opinion good enough for normal use.&lt;/p&gt;
&lt;p&gt;Now that we&amp;rsquo;re talking about byte offsets it&amp;rsquo;s time to pull out our handy dandy header references.
Since we&amp;rsquo;re dealing with IP datagrams embedded in Ethernet frames, let&amp;rsquo;s take a moment to inspect
both headers.&lt;/p&gt;
&lt;p&gt;&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-06-20-misadventures-with-tcpdump-filters/ethernet-frame.gif&#34;
    alt=&#34;Ethernet Frame&#34;&gt;
&lt;/figure&gt;

&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-06-20-misadventures-with-tcpdump-filters/ip_header.gif&#34;
    alt=&#34;IP Datagram Header&#34;&gt;
&lt;/figure&gt;
&lt;/p&gt;
&lt;p&gt;As we can see, the Ethernet frame header is pretty well static, and easily understood. The IP
datagram header is variable length, but the options are all at the end of the header, so for our
purposes today we can consider it a fixed length as well. This makes our calculations very easy and
look something like this.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;This is known to be Ethernet, so add up the length of the header fields (22 bytes) and skip past those.&lt;/li&gt;
&lt;li&gt;Source address in the IP header starts at byte offset 13, so check those 4 bytes.&lt;/li&gt;
&lt;li&gt;Destination address in the IP header starts at byte offset 17, so check those 4 bytes.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;So now we&amp;rsquo;re getting somewhere, and we actually run face first into a surprise. Remember that I said
my example packet capture was taken off an interface that pass the VLAN information. Take another
look at the headers and see if you can identify the field that contains the VLAN tag information.
Hint: You won&amp;rsquo;t because it&amp;rsquo;s not there.&lt;/p&gt;
&lt;p&gt;Enabling VLANs actually do something interesting to your Ethernet frame header. It adds a few extra
fields to your header to a total of 4 bytes. In most cases you won&amp;rsquo;t see this. Generally each switch
port has two modes, access and trunk. An access port is one that you would hand out to a user. This
will get connected directly to a computer or a standard unmanaged mini-switch. A trunk port is extra
special and is often only used either to connect the networking infrastructure or a server that
needs to access several different networks. The extra VLAN header information is only useful on over
a trunk, and as such is stripped out before the frame is transmitted on an access port. So on an
access port, that header doesn&amp;rsquo;t exist, so your dumb byte offset math works pretty well. Remember
at the beginning when I said mirrored ports followed many of the same rules as a trunk? This is
where we begin to see it. Let&amp;rsquo;s now take a look at what happens to the Ethernet frame header when we
add in the VLAN tag information.&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-06-20-misadventures-with-tcpdump-filters/TCPIP_802.1Q.jpg&#34;
    alt=&#34;802.1q Header Layout&#34;&gt;
&lt;/figure&gt;

&lt;p&gt;Knowing that we were doing some dumb parsing by counting byte offsets, and all of our numbers were
based on an Ethernet frame without the VLAN information, we should finally begin to understand our
problem. We are dealing with an off-by-4 byte error. According to our IP header quick reference we
can do a quick offset calculation and see that we&amp;rsquo;re attempting to compare the source address
against the combination of TimeToLive+Type+HeaderChecksum and attempting to compare the destination
address against the source address.&lt;/p&gt;
&lt;p&gt;You should be thinking, &amp;ldquo;Now Scott, yes, this a problem, but we&amp;rsquo;ll still see half of the
communication because when we check for destination address we&amp;rsquo;ll still end up matching against
source!&amp;rdquo; You would be absolutely correct, except for one problem. As I mentioned before the filter
isn&amp;rsquo;t completely as dumb as we&amp;rsquo;re pretending that it is. The base assumption for BPF is that when
you say host, you&amp;rsquo;re talking about an IP address. So the filter does actually check the version
field to see if the packet is IPv4 or IPv6, values 4 or 6 respectively. The IP version field is the
higher order nibble of the first byte. Since we have an off-by-4 byte situation what value are we
actually checking? The answer is the higher order nibble of the third byte in the VLAN header. This
byte contains the 3 bit PCP field and the 1 bit CFI flag. The 3 bit PCP is actually the 802.1p
service priority used in Quality of Service systems.&lt;/p&gt;
&lt;p&gt;In most cases 802.1p is unused, which means a QoS of 0, which means those 3 bits are unset. The 1
bit CFI flag, also called Drop Eligible or DE, is used by PCP to say that in the presence of QoS
based congestion this packet can be dropped. Since 802.1p is generally not used, this field is also
typically 0. In normal situations out filter reads the 0, which is neither a 4 nor a 6, and so our
filter automatically rejects. However, since the priority and DE fields are set by QoS systems we
could have a situation where the filter accidentally works. If ever 802.1p based QoS is used, the
DE flag is unset, and the priority is set to 2 on a scale of 0 (best effort) to 7 (highest) the
filter will still believe we&amp;rsquo;re inspecting an IPv4 packet. Or if the priority is set to 3 and the
DE flag is unset then the filter will believe we&amp;rsquo;re looking at an IPv6 packet. This is all a bit of
an aside since it has been my experience that QoS is rarely used, however it does present an
interesting edge case.&lt;/p&gt;
&lt;p&gt;Ignoring any possibility of QoS in play and going back to straight up 802.1q tagged packets what we
have to do instead is modify the filter string to tell the BPF to treat tagged packets as tagged,
like so:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;tcpdump -nnr capturefile.pcap vlan and host 10.10.15.15
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;What we end up doing here is filtering only for packets containing a VLAN tag and either of the
address fields in the IP header contains 10.10.15.15. By explicitly applying the vlan macro the
filtering system will properly detect the VLAN header and account for it when processing the other
embedded protocols. It is worth noting that this will only match on packets that contain the VLAN
header. If you want to generalize your filter, say you don&amp;rsquo;t know or your capture contains a mix of
packets that may or may not have a VLAN tag, you can complicate your filter to do something like
this.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;tcpdump -nnr capturefile.pcap &#39;host 10.10.15.15 or ( vlan and host 10.10.15.15 )&#39;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Finding out that VLANs are used on networks that you&amp;rsquo;re dealing with, and if the infrastructure is
any more complicated than a 10 person office it probably does, has some pretty far reaching
consequences. Any time one applies pcap filters to a capture you&amp;rsquo;ll need to take into account
802.11q tags. You&amp;rsquo;ll definitely want to keep this in mind when using BPF files to distribute load
across multiple snort processes or when using BPFs to do targeted analysis using tools like Argus.
Depending on the configuration of your interface, your monitoring port may actually have a native
vlan. If that is the case you&amp;rsquo;ll find that you do receive data, which may disguise the fact that
you&amp;rsquo;re not receiving all of the data.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>Base Rulesets in IPTables</title>
      <link>https://secopsmonkey.com/post/2014-06-06-base-rulesets-in-iptables/</link>
      <pubDate>Fri, 06 Jun 2014 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-06-06-base-rulesets-in-iptables/</guid>
      
        <description>&lt;blockquote&gt;
&lt;p&gt;Pulling back from the archives this is a repost of a previous blog post. This time ripped from a
guest spot at &lt;a href=&#34;http://security.blogoverflow.com/2011/08/base-rulesets-in-iptables/&#34;&gt;The Security Stack Exchange Community Blog&lt;/a&gt;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;This &lt;a href=&#34;http://security.stackexchange.com/q/4150/618&#34;&gt;question&lt;/a&gt; on &lt;a href=&#34;http://security.stackexchange.com/&#34;&gt;Security.SE&lt;/a&gt; made me think in a rather devious way. At first, I found it
to be rather poorly worded, imprecise, and potentially not worth salvaging. After a couple of days I
started to realize exactly how many times I&amp;rsquo;ve really been asked this question by well intentioned,
and often, knowledgeable people. The real question should be, &amp;ldquo;Is there a recommended set of firewall
rules that can be used as a standard config?&amp;rdquo; Or more plainly, I have a bunch of systems, so what
rules should they all have no matter what services they provide. Now that is a question that can
reasonably be answered, and what I&amp;rsquo;ve typically given is this:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp --icmp-type any -j ACCEPT
# Force SYN checks
-A INPUT -p tcp ! --syn -m state --state NEW -j DROP
# Drop all fragments
-A INPUT -f -j DROP
# Drop XMAS packets
-A INPUT -p tcp --tcp-flags ALL ALL -j DROP
# Drop NULL packets
-A INPUT -p tcp --tcp-flags ALL NONE -j DROP
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The first and last lines should be pretty obvious so I won&amp;rsquo;t go into too much detail. Enough
services use loopback that, except in very restrictive environments, attempting to firewall them
will almost definitely be more work than is useful. Similarly, the ESTABLISHED keyword allows return
traffic for outgoing connections. The RELATED keyword takes care of things like FTP that use multiple
ports and may trigger multiple flows that are not part of the same connection, but are none-the-less
dependent on each other. In a perfect world these would be at the top for performance reasons. Since
rules are processed in order we really want the fewest number of rules processed as possible,
however in order to get the full benefit from the above rule set, we want to run as many packets as
possible by them.&lt;/p&gt;
&lt;p&gt;The allow all on ICMP is probably the most controversial part of this set. While there are
reconnaissance concerns with ICMP, the network infrastructure is designed with the assumption that
ICMP is passed. You&amp;rsquo;re better off allowing it (at least within your organization&amp;rsquo;s network space),
or grokking all of the ICMP messages and determining your own balance. Look at this &lt;a href=&#34;http://en.wikipedia.org/wiki/ICMP_Destination_Unreachable&#34;&gt;question&lt;/a&gt;
for some worthwhile discussion on the matter.&lt;/p&gt;
&lt;p&gt;Now, down to the brass tacks. Let&amp;rsquo;s look at each of the wonky rules in order.&lt;/p&gt;
&lt;h1 id=&#34;forcing-syn-checks&#34;&gt;Forcing SYN Checks&lt;/h1&gt;
&lt;pre&gt;&lt;code&gt;-A INPUT -p tcp ! --syn -m state --state NEW -j DROP
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This rule performs two checks:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Is the SYN bit NOT set, and&lt;/li&gt;
&lt;li&gt;Is this packet NOT part of a connection in the state table&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;If both conditions match, then the packet gets dropped. This is a bit of a low-hanging fruit kind of
rule. If these conditions match, then we&amp;rsquo;re looking at a packet that we just downright shouldn&amp;rsquo;t be
interested in. This could indicate a situation where the connection has been pruned from the
conntrack state table, or perhaps a malicious replay event. Either way, there isn&amp;rsquo;t any typical
benefit to allowing this traffic, so let&amp;rsquo;s explicitly block it.&lt;/p&gt;
&lt;h1 id=&#34;fragments-be-damned&#34;&gt;Fragments Be Damned&lt;/h1&gt;
&lt;pre&gt;&lt;code&gt;-A INPUT -f -j DROP
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This is an easy one. Drop any packet with the fragment bit set. I fully realize this sounds pretty
severe. Networks were designed with the notion of fragmentation, in fact the the IPv4 header
specifically contains a flag that indicates whether or not that packet should or should not be
fragmented. Considering that this is a core feature of IPv4, fragmentation is still a bit of a
touchy subject. If the DF bit is set, then MTU path discovery should just work. However, not all
devices respond back with the correct ICMP message. The above rule is one of them, however that&amp;rsquo;s
because ICMP type 3 code 4 (&lt;a href=&#34;http://security.stackexchange.com/q/4440/618&#34;&gt;wikipedia&lt;/a&gt;) isn&amp;rsquo;t a reject option in iptables. As a result of this
one can&amp;rsquo;t really know whether or not your packets will get fragmented along the network. Nowadays,
on internal networks at least, this usually isn&amp;rsquo;t a problem. You may run into problems, however,
when dealing with VPNs and similar where your 1500 byte ethernet segment suddenly needs to make
space for an extra header.&lt;/p&gt;
&lt;p&gt;So now that we&amp;rsquo;ve talked about all the reasons to not drop fragments, here&amp;rsquo;s the reason to. By
default, standard iptables rules are only applied to the packet marked as the first fragment.
Meaning, any packet marked as a fragment with an offset of 2 or greater is passed through, the
assumption being that if we receive an invalid packet then reassembly will fail and the packets will
get dropped anyway. In my experience, fragmentation is a small enough problem that I don&amp;rsquo;t want to
deal with risk and block it anyway. This should only get better with IPv6 as path MTU discovery is
placed more firmly on the client and is considered less &amp;ldquo;Magic.&amp;rdquo;&lt;/p&gt;
&lt;h1 id=&#34;christmas-in-july&#34;&gt;Christmas in July&lt;/h1&gt;
&lt;pre&gt;&lt;code&gt;-A INPUT -p tcp --tcp-flags ALL ALL -j DROP
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Network reconnaissance is a big deal. It allows us to get a good feel for what&amp;rsquo;s out there so that
when doing our work we have some indications of what might exist instead of just blindly stabbing in
the dark. So called ‘Christmas Tree Packets&amp;rsquo; are one of those reconnaissance techniques used by most
network scanners. The idea is that for whatever protocol we use, whether TCP/UDP/ICMP/etc, every
flag is set and every option is enabled. The notion is that just like a old style indicator board
the packet is &amp;ldquo;lit up like a Christmas tree&amp;rdquo;. By using a packet like this we can look at the
behavior of the responses and make some guesses about what operating system and version the remote
host is running. For a White Hat, we can use that information to build out a distribution graph of
what types of systems we have, what versions they&amp;rsquo;re running, where we might want to focus our
protections, or who we might need to visit for an upgrade/remediation. For a Black Hat, this
information can be used to find areas of the network to focus their attacks on or particularly
vulnerable looking systems that they can attempt to exploit. By design some flags are incompatible
with each-other and as a result any Christmas Tree Packet is at best a protocol anomaly, and at
worst a precursor to malicious activity. In either case, there is normally no compelling reason to
accept such packets, so we should drop them just to be safe.&lt;/p&gt;
&lt;p&gt;I have read instances of Christmas Tree Packets resulting in Denial of Service situations,
particularly with networking gear. The idea being that since so many flags and options are set, the
processing complexity, and thus time, is increased. Flood a network with these and watch the router
stop processing normal packets. In truth, I do not have experience with this failure scenario.&lt;/p&gt;
&lt;h1 id=&#34;nothing-to-see-here&#34;&gt;Nothing to See Here&lt;/h1&gt;
&lt;pre&gt;&lt;code&gt;-A INPUT -p tcp --tcp-flags ALL NONE -j DROP
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;When we see a packet where none of the flags or options are set we use the term Null Packet. Just
like with the above Christmas Tree Packet, one should not see this on a normal, well behaved
network. Also, just as above, they can be used for reconnaissance purposes to try and determine the
OS of the remote host.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>Fighting NFS Mounts at Boot Time</title>
      <link>https://secopsmonkey.com/post/2014-05-30-fighting-nfs-mounts-at-boot-time/</link>
      <pubDate>Fri, 30 May 2014 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-05-30-fighting-nfs-mounts-at-boot-time/</guid>
      
        <description>&lt;blockquote&gt;
&lt;p&gt;Pulling back from the archives this is a repost of a previous blog post. This time ripped from a
guest spot at &lt;a href=&#34;http://thenubbyadmin.com/2013/04/10/solving-nfs-mounts-at-boot-time/&#34;&gt;The Nubby Admin&lt;/a&gt;, a fantastic blog from a fellow tech nerd.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Let&amp;rsquo;s face it. NFS is a magical thing. It allows you to centralize your storage,
share volumes across systems, and all while maintaining sane permissions and
ownership. Unfortunately, it can also be a bit of a fickle beast. Let&amp;rsquo;s say you
just had your volume configured and you set up the mounts. You go and run this
command:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;mount -t nfs 10.10.10.1:/vol1/fs1 /data
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Works like a champ, you now have your data partition mounted over NFS. So you
add this line to your &lt;code&gt;/etc/fstab&lt;/code&gt; and make it mount automagically.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;10.10.10.1:/vol1/fs1  /data             nfs     defaults    0 0
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;A few weeks go by and you apply a kernel update. No big deal, you apply the
updates and during your next maintenance window reboot to apply the new kernel.
Then you start to see applications failing and notice the volume isn&amp;rsquo;t actually
mounted. This is an unfortunate result of the automounter subsystem.&lt;/p&gt;
&lt;p&gt;It&amp;rsquo;s like this. At boot time the root partition gets mounted, automounter reads
the &lt;code&gt;/etc/fstab&lt;/code&gt; file, and boots any filesystem that doesn&amp;rsquo;t have &lt;code&gt;noauto&lt;/code&gt; as a
mount option. Since we&amp;rsquo;re still very early in the boot process the network isn&amp;rsquo;t
up yet, so naturally any network filesystems fail. The real problem here is that
at no point does automounter go back and attempt to remount those systems. So
your NFS mount points fail because there is no network, and done is done.&lt;/p&gt;
&lt;p&gt;The developers were nice enough to provide a fix for this. There exists a mount
option called &lt;code&gt;_netdev&lt;/code&gt;. If we quote directly from the man page (sourced from
RHEL 6.4):&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;_netdev
        The  filesystem  resides  on  a device that requires network access (used to prevent the system from attempting to mount these filesystems until the
        network has been enabled on the system).
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This is awesome, and exactly what we want. So you modify your entry in fstab to
look like this:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;10.10.10.1:/vol1/fs1  /data             nfs     defaults,_netdev    0 0
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;You&amp;rsquo;ve been bitten by NFS mounting in the past so you throw this in your test
environment and reboot immediately. After the system comes up you notice a
problem. Your NFS volumes are still unmounted. You see, there&amp;rsquo;s a bit of a
hitch. Automounter followed the same procedure that it did before, except this
time it didn&amp;rsquo;t even attempt to mount &lt;code&gt;/data&lt;/code&gt;. The &lt;code&gt;_netdev&lt;/code&gt; option doesn&amp;rsquo;t tell
the system to mount the filesystem when network comes up, it says don&amp;rsquo;t attempt
to mount it at all if the network isn&amp;rsquo;t up. There is still a missing piece to
the puzzle. If you look at your init scripts there is a service called &lt;code&gt;netfs&lt;/code&gt;.
If you read the script you can see in the chkconfig header this description:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;# description: Mounts and unmounts all Network File System (NFS), \
#              CIFS (Lan Manager/Windows), and NCP (NetWare) mount points.
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This is exactly what you need. It is a service whose sole purpose is to read
your &lt;code&gt;/etc/fstab&lt;/code&gt; and mount network filesystems. All you have to do is enable it&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;chkconfig netfs on
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;and watch the magic happen. Now your mount boot process should look something
like this:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Automounter reads &lt;code&gt;/etc/fstab&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Ignores &lt;code&gt;/data&lt;/code&gt; since it has &lt;code&gt;_netdev&lt;/code&gt; option set&lt;/li&gt;
&lt;li&gt;Mounts all other filesystems&lt;/li&gt;
&lt;li&gt;Finishes mount jobs and allows system to continue booting&lt;/li&gt;
&lt;li&gt;Network comes up&lt;/li&gt;
&lt;li&gt;Service &lt;code&gt;netfs&lt;/code&gt; started&lt;/li&gt;
&lt;li&gt;netfs reads &lt;code&gt;/etc/fstab&lt;/code&gt; and finds an nfs filesystem&lt;/li&gt;
&lt;li&gt;netfs mounts &lt;code&gt;/data&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;What&amp;rsquo;s funny is that while I was researching this problem I never stumbled
across netfs as a service. I had even gone so far as to start planning out my
own custom init script that would do exactly this, except specifically for my
mount points instead of generalizing. It&amp;rsquo;s nice to see that I was on the right
track, but even better that the tools already existed.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>Why is Change Management Hard?</title>
      <link>https://secopsmonkey.com/post/2014-05-22-why-is-change-management-hard/</link>
      <pubDate>Thu, 22 May 2014 12:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-05-22-why-is-change-management-hard/</guid>
      
        <description>&lt;p&gt;No matter how much we hope otherwise the foundation of &lt;em&gt;any&lt;/em&gt; security program
are consistent and used procedures. This means figuring out what we need to be
doing, sketching out how we think we should be doing it, finding out we were
totally wrong and misguided thinking we could do it that way, then editing it
into something that actually works.&lt;/p&gt;
&lt;p&gt;I knew walking into this job that procedure&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a href=&#34;#fn:1&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt; generation like this would be a
necessary part of my job. After I got settled with the various aspects of the
company enough to know what&amp;rsquo;s going on I sat down to work on generating some of
those documents. I banged out a couple without to much effort before landing on
the topic of Change Management.&lt;/p&gt;
&lt;p&gt;I really struggled with this one. In order for change control to work there must
be a certain amount of formality. I&amp;rsquo;ve participated in environments where
scheduling a change consisted of shouting out your office door, &amp;ldquo;Wiki is going
down for maintenance upgrades!&amp;rdquo; This breaks down because someone might not hear
you and 3 months from now when we discover backups had stopped working nobody
will remember who did the work or when or whether the backup process had been
checked. Too much formality can be just as bad. If the completely formal fully
ITIL compliance change process requires scheduling a maintenance window 4 weeks
in advance, rollout plan, rollback plan, test plan, peer review, and customer
then your admin will be pretty bent when upgrading the wiki consists of running
&lt;code&gt;yum update -y wiki-soft&lt;/code&gt;. In this case don&amp;rsquo;t be surprised if your entire
process gets ignored for anything except super scary work.&lt;/p&gt;
&lt;p&gt;Finding the right balance is really tough because, quite honestly, the level of
formality depends on the overall risk impact of the change compared against the
risk appetite of your business. However, writing your process in such a way as
to account for differing levels of formality results in a very formal process!&lt;/p&gt;
&lt;p&gt;Trying to walk this balance is tricky and is really making me think hard about
what a Change Management program is trying to accomplish. At the core I finally
settled on:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Limit the risk of conflicting work, i.e. multiple people doing different
things at the same time that would affect each other.&lt;/li&gt;
&lt;li&gt;Document what happened when so we can more easily troubleshoot problems.&lt;/li&gt;
&lt;li&gt;Know when things are going to happen.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The benefits of the first two are pretty obvious to anyone who&amp;rsquo;s worked in big
shops before, the bigger the shop the worse it becomes. The last one is more
about a holistic communication model and is primarily focused on the customers.
If we know a service will go down for 45 minutes on Thursday we can shoot an
email out to our clients or post it on a web page, this way if they try to use
the service and it doesn&amp;rsquo;t work they don&amp;rsquo;t get freaked out. Or even more
importantly the helpdesk, who are normally completely left out of things like
this, can answer calls with a soothing and confident, &amp;ldquo;Our engineers are
performing maintenance on that service, they expect to be finished within 15
minutes.&amp;rdquo;&lt;/p&gt;
&lt;p&gt;Fortunately we can address all those pretty informally since it all comes down
to who does what when. Personally I lean a little more towards that end of the
spectrum but the risk averse side of me really digs the comfort of a good roll
out plan. I still haven&amp;rsquo;t fully baked what we need but the more it stews the
more I&amp;rsquo;m convinced that for &lt;em&gt;my&lt;/em&gt; team less formality is better so as to remain
more flexible. We aren&amp;rsquo;t what I would call a properly Agile office but have
definitely adapted some of those values to our work.&lt;/p&gt;
&lt;p&gt;I would love to hear stories of how people have made change management work in
smaller teams. Leave comments below or that new fangled Twitter thingamajigger.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;I should really be calling this policy. I&amp;rsquo;m still purging myself of the
Pavlovian responses drilled into me as a public employee so it&amp;rsquo;s still difficult
to say the word &amp;ldquo;policy&amp;rdquo; without thinking of a multi-year draft-&amp;gt;publication
process that involves as much horse-trading as a typical Senate bill.&amp;#160;&lt;a href=&#34;#fnref:1&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>A Brief Introduction to auditd</title>
      <link>https://secopsmonkey.com/post/2014-05-16-a-brief-introduction-to-auditd/</link>
      <pubDate>Fri, 16 May 2014 12:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-05-16-a-brief-introduction-to-auditd/</guid>
      
        <description>&lt;p&gt;&lt;em&gt;Pulling back from the archives this is a repost of a previous blog post. This time ripped from a
guest spot at &lt;a href=&#34;http://security.blogoverflow.com/2013/01/a-brief-introduction-to-auditd/&#34;&gt;The Security Stack Exchange Community Blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;The &lt;a href=&#34;http://people.redhat.com/sgrubb/audit/&#34;&gt;auditd&lt;/a&gt; subsystem is an access monitoring and accounting for Linux developed and maintained
by RedHat. It was designed to integrate pretty tightly with the kernel and watch for interesting
system calls. Additionally, likely because of this level of integration and detailed logging, it is
used as the logger for SELinux.&lt;/p&gt;
&lt;p&gt;All in all, it is a pretty fantastic tool for monitoring what’s happening on your system. Since it
operates at the kernel level this gives us a hook into any system operation we want. We have the
option to write a log any time a particular system call happens, whether that be &lt;code&gt;unlink&lt;/code&gt; or &lt;code&gt;getpid&lt;/code&gt;.
We can monitor access to any file, all network traffic, really anything we want. The level of
detail is pretty phenomenal and, since it operates at such a low level, the granularity of
information is incredibly useful.&lt;/p&gt;
&lt;p&gt;The biggest downfall is actually a result of the design that makes it so handy. This is itself a
logging system and as a result does not use syslog. The good thing here is that it doesn’t have to
rely on anything external to operate, so a typo in your (syslog|rsyslog|syslog-ng).conf file won’t
result in losing your system audit logs. As a result you’ll have to manage all the audit logging
using the auditd suite of tools. This means any kind of log collection, organization, or archiving
may not work with these files, including remote logging. As an aside, auditd does have provisions
for remote logging, however they are not as trivial as we’ve come to expect from syslog.&lt;/p&gt;
&lt;p&gt;Thanks to the level of integration that it provides your auditd configurations can be quite complex,
but I’ve found that there are primarily only two options you need to know.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;-a exit,always -S &amp;lt;syscall&amp;gt;&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;-w &amp;lt;filename&amp;gt;&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The first of these generates a log whenever the listed syslog exits, and whenever the listed file is
modified. Seems pretty easy right? It certainly can be, but it does require some investigation into
what system calls interest you, particularly if you’re not familiar with OS programming or POSIX.
Fortunately for us there are some standards that give us some guidance on what to look out for.
Let’s take, for example, the [Center for Internet Security] [Red Hat Enterprise Linux 6 Benchmark].
The relevant section is “5.2 Configure System Account (auditd)” starting on page 99. There is a
large number of interesting examples listed, but for our purposes we’ll whittle those down to a more
minimal and assume your &lt;code&gt;/etc/audit/audit.rules&lt;/code&gt; looks like this.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.
# First rule - delete all
-D

# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 1024
-a always,exit -S adjtimex -S settimeofday -S stime -k time-change
-a always,exit -S clock_settime -k time-change
-a always,exit -S sethostname -S setdomainname -k system-locale
-w /etc/group -p wa -k identity
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k identity
-w /var/run/utmp -p wa -k session
-w /var/log/wtmp -p wa -k session
-w /var/log/btmp -p wa -k session
-w /etc/selinux/ -p wa -k MAC-policy
# Disable adding any additional rules - note that adding new rules will require a reboot
-e 2
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Based on our earlier discussion we should be able to see that we generate a log message every time
any of the following system calls exit: &lt;code&gt;adjtimex&lt;/code&gt;, &lt;code&gt;settimeofday&lt;/code&gt;, &lt;code&gt;stime&lt;/code&gt;, &lt;code&gt;clock_settime&lt;/code&gt;,
&lt;code&gt;sethostname&lt;/code&gt;, &lt;code&gt;setdomainname&lt;/code&gt;. This will let us know whenever the time gets changed or if the host
or domain name of the system get changed.&lt;/p&gt;
&lt;p&gt;We’re also watching a few files. The first four (group, passwd, shadow, sudo) will let us know
whenever users get added, modified, or privileges changed. The next three files (utmp, wtmp, btmp)
store the current login state of each user, login/logout history, and failed login attempts
respectively. So monitoring these will let us know any time an account is used, or failed login
attempt, or more specifically whenever these files get changed which will include malicious covering
of tracks. Lastly, we’re watching the directory &lt;code&gt;/etc/selinux/&lt;/code&gt;. Directories are a special case in
that this will cause the system to recursively monitor the files in that directory. There is a
special caveat that you cannot watch &lt;code&gt;/&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;When watching files we also added the option &lt;code&gt;-p wa&lt;/code&gt;. This tells auditd to only watch for (w)rites
or (a)ttribute changes. It should be noted that for write (and read for that matter) we aren’
t actually logging on those system calls. Instead we’re logging on &lt;code&gt;open&lt;/code&gt; if the appropriate flags
are set.&lt;/p&gt;
&lt;p&gt;It should also be said that the logs are also rather&amp;hellip;complete. As an example I added the system
call rule for &lt;code&gt;sethostname&lt;/code&gt; to a Fedora 17 system, with audit version 2.2.1. This is the resultant
log from running &lt;code&gt;hostname audit-test.home.private&lt;/code&gt; as root.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;type=SYSCALL msg=audit(1358306046.744:260): arch=c000003e syscall=170 success=yes exit=0 a0=2025010 a1=17 a2=7 a3=18 items=0 ppid=23922 pid=26742 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts4 ses=16 comm=&amp;quot;hostname&amp;quot; exe=&amp;quot;/usr/bin/hostname&amp;quot; subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=&amp;quot;system-locale&amp;quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;There are gobs of fields listed, however the ones that interest me the most are the various field
names containing the letters “id”, “exe” and that ugly string of numbers in the first parens. The
first bit, 1358306046.744, is the timestamp of the event in epoch time. The exe field contains the
full path to the binary that was executed. Useful, since we know what was run, but it does not
contain the full command line including arguments. Not ideal.&lt;/p&gt;
&lt;p&gt;Next we see that the command was run by root, since the euid is 0. Interestingly, the field auid
(called audit uid) contains 1000, which is the uid of my regular user account on that host. The auid
field actually contains the user id of the original logged in user for this login session. This
means, that even though I used &lt;code&gt;su -&lt;/code&gt; to gain a root shell the auditing subsystem still knows who I
am. Using su to gain a root shell has always been the bane of account auditing, but the auditd
system records information to usefully identify a user. It does not forgive the lack of command line
options, but certainly makes me feel better about it.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>Custom Partitioning using Preseed</title>
      <link>https://secopsmonkey.com/post/2014-05-09-custom-partioning-using-preseed/</link>
      <pubDate>Fri, 09 May 2014 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-05-09-custom-partioning-using-preseed/</guid>
      
        <description>&lt;h1 id=&#34;simplistic-beginnings&#34;&gt;Simplistic Beginnings&lt;/h1&gt;
&lt;p&gt;Whether using a preseed or a manual install the default partitioning setup is to use what Debian
calls &amp;ldquo;atomic&amp;rdquo;. When using atomic the installer creates two partitions&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;/boot&lt;/code&gt; at 250MB&lt;/li&gt;
&lt;li&gt;LVM physical volume (PV) filled to rest of disk&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The one PV contains a single Volume Group (VG) which will contain two Logical Volumes (LV).&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;$(hostname)-vg/swap_1&lt;/code&gt; at the same size as your RAM&lt;/li&gt;
&lt;li&gt;&lt;code&gt;$(hostname)-vg/root&lt;/code&gt; mounted on &lt;code&gt;/&lt;/code&gt; filled to rest of VG&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The logic is a little deeper; while swap defaults to the same size as the RAM you have it will
expand up to twice the RAM. Similarly &lt;code&gt;/&lt;/code&gt; will be at least 500MB and will expand out to fill the
rest of the partition.&lt;/p&gt;
&lt;p&gt;This is quick, easy, you don&amp;rsquo;t have to worry about partitions filling up, and all together pretty
scary. I&amp;rsquo;ll save you the rant by linking to the &lt;a href=&#34;http://serverfault.com/a/449580/3356&#34;&gt;ServerFault answer&lt;/a&gt; but I&amp;rsquo;ve written about some
of the dangers of single partitions in the past. Suffice to say separating your system out into
multiple partitions is handy and allows us to have an overall more resilient and secure system.&lt;/p&gt;
&lt;h1 id=&#34;baby-steps-into-complexity&#34;&gt;Baby Steps into Complexity&lt;/h1&gt;
&lt;p&gt;Debian has a preseed notion that&amp;rsquo;s called &amp;ldquo;multi&amp;rdquo;. Just like &amp;ldquo;atomic&amp;rdquo; it will automatically create
two partitions, but the real changes happen within the VG itself. Instead of the two LVs above we&amp;rsquo;ll
now have 6 arranged thusly.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;$(hostname)-vg/swap_1&lt;/code&gt; at 96MB up to twice size of installed RAM&lt;/li&gt;
&lt;li&gt;&lt;code&gt;$(hostname)-vg/root&lt;/code&gt; mounted on &lt;code&gt;/&lt;/code&gt; minimum of 300MB up to 350MB&lt;/li&gt;
&lt;li&gt;&lt;code&gt;$(hostname)-vg/usr&lt;/code&gt; mounted on &lt;code&gt;/usr/&lt;/code&gt; minimum of 500MB up to 9000MB&lt;/li&gt;
&lt;li&gt;&lt;code&gt;$(hostname)-vg/var&lt;/code&gt; mounted on &lt;code&gt;/var/&lt;/code&gt; minimum of 300MB up to 3000MB&lt;/li&gt;
&lt;li&gt;&lt;code&gt;$(hostname)-vg/tmp&lt;/code&gt; mounted on &lt;code&gt;/tmp/&lt;/code&gt; minimum of 20MB up to 400MB&lt;/li&gt;
&lt;li&gt;&lt;code&gt;$(hostname)-vg/home&lt;/code&gt; mounted on &lt;code&gt;/home/&lt;/code&gt; minimum of 300MB up to remaining size&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;How the partitions resize gets a little weird. As we can see each partition has a minimum size and a
maximum size, additionally each partition has what&amp;rsquo;s called a priority where priority is a number
between 1 and big. During preseeding the minimum, maximum, and priority are all fed into a formula,
given below, to determine the actual allocated size. All in all it&amp;rsquo;s an interesting system but the
formula could be more straightforward and the defaults chosen above seen much too small for my
tastes. This means I need to dig in and craft my own.&lt;/p&gt;
&lt;h1 id=&#34;building-your-own&#34;&gt;Building Your Own&lt;/h1&gt;
&lt;p&gt;If you&amp;rsquo;ll recall from some previous posts I&amp;rsquo;ve been getting Cobbler running to handle all my
provisioning which is both good and annoying here. Good in that I can move the partitioning to a
snippet, making it easy to customize for a specific host. Annoying in that the syntax is a little
janky and requires some extra considerations for the Cheetah interpreter. To see what I mean let&amp;rsquo;s
look at a single partition example.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;d-i partman-auto/expert_recipe string         \
    4000 800 8000  $default_filesystem        \
        $lvmok{ }                             \
        method{ format }                      \
        format{ }                             \
        use_filesystem{ }                     \
        $default_filesystem{ }                \
        mountpoint{ / }                       \
    .
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The first line tells preseed that what follows is a custom partition config. Then for each partition
we have to create a block that looks like the one above. Since that block contains &lt;code&gt;$lvmok&lt;/code&gt; it will
be a Logical Volume (LV) in the default VG, since we didn&amp;rsquo;t specify one. The LV will be formatted
using whatever the standard filesystem is, most likely ext3, and the &lt;code&gt;/etc/fstab&lt;/code&gt; file will be
configured to mount it at &lt;code&gt;/&lt;/code&gt;. Notice the string of numbers at the beginning? Those are the minimum
size (4000MB), the priority (800), and the maximum size (8000MB).&lt;/p&gt;
&lt;p&gt;The Cheetah problem comes into play when we look at the variables. Since I&amp;rsquo;m using Cobbler the
preseed will be passed through an interpreting engine so I have to treat it a little differently,
much like a Puppet template file only with less explicit code. When we request the preseed file
from Cobbler it will pass it through the Cheetah interpreter which will process all the linked
files and pass them back up to Cobbler for display. This means the variables, &lt;code&gt;$default_filesystem&lt;/code&gt;
and &lt;code&gt;$lvmok&lt;/code&gt;, will be evaluated as Cheetah code and almost definitely fail. The fix is easy, albeit
annoying. We have to escape the dollar signs so Cheetah leaves them alone and lets &lt;code&gt;debian-installer&lt;/code&gt;
handle them at install time.&lt;/p&gt;
&lt;h1 id=&#34;partioning-guidance&#34;&gt;Partioning Guidance&lt;/h1&gt;
&lt;p&gt;Now that we&amp;rsquo;ve figured out exactly how to construct our scheme we need to decide what our partitions
are going to be. My typical guidance has been to create, at minimum&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;/boot&lt;/li&gt;
&lt;li&gt;/&lt;/li&gt;
&lt;li&gt;/home&lt;/li&gt;
&lt;li&gt;/tmp&lt;/li&gt;
&lt;li&gt;/var&lt;/li&gt;
&lt;li&gt;/var/log/audit&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This gives you some logical separation of primary components while keeping things rather simple.
I&amp;rsquo;ve always been rather happy with the hardening guides produced by the &lt;a href=&#34;http://benchmarks.cisecurity.org/downloads/browse/&#34;&gt;Center for Internet Security&lt;/a&gt;.
Their OS guides will typically have partitioning guidance, as well as some specialized mount
options. What they don&amp;rsquo;t do is give you an idea as to what size to use for each partition. In the
RedHat world I usually recommended&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;# Mount Point       Min Size (MB)    Max Size (MB)
/                   4000             8000
/home               1000             4000
/tmp                1000             2000
/var                2000             4000
swap                1000             2000
/var/log/audit       250
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I&amp;rsquo;m still getting a feel for how Ubuntu does things so I don&amp;rsquo;t yet know if those sizes are really
the best idea. I suspect they will be but time will tell.&lt;/p&gt;
&lt;h1 id=&#34;creating-the-snippet&#34;&gt;Creating the Snippet&lt;/h1&gt;
&lt;p&gt;Combining all that information we can concoct our partitioning snippet. The trick, of course, is
getting the priority values right. I&amp;rsquo;m not convinced they&amp;rsquo;re correct yet, since my &lt;code&gt;/&lt;/code&gt; partition was
less than max but my swap was max. I find that a bit less than ideal so put some effort into
figuring out if different priorities work better. If you find the right ones, please let me know. I
have to admit that the thought of tweaking those sounds painful.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;# Disk Partitioning
# Use LVM, and wipe out anything that already exists
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman-auto/method string lvm
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-auto-lvm/new_vg_name string system
d-i partman-md/device_remove_md boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/default_filesystem string ext4

d-i partman-auto/expert_recipe string     \
  multi-cnx ::                            \
    128 512 256 ext2                      \
    \$defaultignore{ }                    \
    method{ format }                      \
    format{ }                             \
    use_filesystem{ }                     \
    filesystem{ ext2 }                    \
    mountpoint{ /boot }                   \
    .                                     \
    4000 800 8000  \$default_filesystem   \
    \$lvmok{ }                            \
    method{ format }                      \
    format{ }                             \
    use_filesystem{ }                     \
    \$default_filesystem{ }               \
    mountpoint{ / }                       \
    .                                     \
    2000 1500 4000 \$default_filesystem   \
    \$lvmok{ }                            \
    method{ format }                      \
    format{ }                             \
    use_filesystem{ }                     \
    \$default_filesystem{ }               \
    mountpoint{ /var }                    \
    .                                     \
    96 512 200% linux-swap                \
    \$lvmok{ }                            \
    \$reusemethod{ }                      \
    method{ swap }                        \
    format{ }                             \
    .                                     \
    1000 300 2000 \$default_filesystem    \
    \$lvmok{ }                            \
    method{ format }                      \
    format{ }                             \
    use_filesystem{ }                     \
    \$default_filesystem{ }               \
    mountpoint{ /tmp }                    \
    .                                     \
    1000 3000 8000 \$default_filesystem   \
    \$lvmok{ }                            \
    method{ format }                      \
    format{ }                             \
    use_filesystem{ }                     \
    \$default_filesystem{ }               \
    mountpoint{ /home }                   \
    .                                     
&lt;/code&gt;&lt;/pre&gt;
&lt;h1 id=&#34;priority-based-partition-size-formula&#34;&gt;Priority Based Partition Size Formula&lt;/h1&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;k&#34;&gt;for&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;1&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;&amp;lt;=&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;N&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;++&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;n&#34;&gt;factor&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;priority&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;-&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;min&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;n&#34;&gt;ready&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;no&#34;&gt;FALSE&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;k&#34;&gt;while&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;!&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;ready&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;n&#34;&gt;minsum&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;min&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;1&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;+&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;min&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;2&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;+&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;...&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;+&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;min&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;N&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;n&#34;&gt;factsum&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;factor&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;1&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;+&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;factor&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;2&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;+&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;...&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;+&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;factor&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;N&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;n&#34;&gt;ready&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;no&#34;&gt;TRUE&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;k&#34;&gt;for&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;=&lt;/span&gt;&lt;span class=&#34;mi&#34;&gt;1&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;&amp;lt;=&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;N&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;++&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;        &lt;span class=&#34;n&#34;&gt;x&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;min&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;+&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;free_space&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;-&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;minsum&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;*&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;factor&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;/&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;factsum&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;        &lt;span class=&#34;k&#34;&gt;if&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;x&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;max&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;          &lt;span class=&#34;n&#34;&gt;x&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;max&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;        &lt;span class=&#34;k&#34;&gt;if&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;(&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;x&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;!=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;min&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;)&lt;/span&gt; &lt;span class=&#34;p&#34;&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;          &lt;span class=&#34;n&#34;&gt;ready&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;no&#34;&gt;FALSE&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;          &lt;span class=&#34;n&#34;&gt;min&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;[&lt;/span&gt;&lt;span class=&#34;n&#34;&gt;i&lt;/span&gt;&lt;span class=&#34;o&#34;&gt;]&lt;/span&gt; &lt;span class=&#34;o&#34;&gt;=&lt;/span&gt; &lt;span class=&#34;n&#34;&gt;x&lt;/span&gt;&lt;span class=&#34;p&#34;&gt;;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;        &lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      &lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;    &lt;span class=&#34;p&#34;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>Windows Time Synchronization The Battle Continues</title>
      <link>https://secopsmonkey.com/post/2014-05-06-windows-time-synchonization-the-battle-continues/</link>
      <pubDate>Tue, 06 May 2014 08:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-05-06-windows-time-synchonization-the-battle-continues/</guid>
      
        <description>&lt;p&gt;While trying to implement the time changes discussed &lt;a href=&#34;http://secopsmonkey.com/time-synchronization-in-windows.html&#34;&gt;earlier&lt;/a&gt; I discovered that Windows time
is a bit more complicated than I at first believed. Let&amp;rsquo;s recap my requirements:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Keep time synchronized from a network service&lt;/li&gt;
&lt;li&gt;Always set the time no matter how far off it is&lt;/li&gt;
&lt;li&gt;Set the time on system startup&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;It seems that requirements 1 and 2 are discussed last time and should be easily set using registry
keys. What I discovered during implementation is that while ignoring the 15 hour drift limit is easy
actual ongoing time synchronization is disabled by default on systems that are not domain joined.&lt;/p&gt;
&lt;p&gt;In Windows all NTP work is handled using the &lt;code&gt;W32Time&lt;/code&gt; service. By default the service is set to
start automatically and, as shown, has no dependencies.&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-05-06-windows-time-synchonization-the-battle-continues/Windows_Time_Service_Dependencies.jpg&#34;
    alt=&#34;Time Service Dependencies&#34;&gt;
&lt;/figure&gt;

&lt;p&gt;This would indicate to me that the service will launch on system start-up. Unfortunately I was quite
wrong. In addition to the service dependencies each service has what&amp;rsquo;s called a &amp;ldquo;trigger&amp;rdquo;. When a
service is set to start &amp;ldquo;Automatically&amp;rdquo; that doesn&amp;rsquo;t mean &amp;ldquo;On Boot&amp;rdquo;, instead it means &amp;ldquo;When My
Trigger Occurs&amp;rdquo;. I&amp;rsquo;ve never dealt with Windows services outside of the snap-in, where the triggers
are not shown, so needless to say I was slightly perturbed to find this out. The W32Time service&amp;rsquo;s
trigger is set to be &amp;ldquo;Domain Join&amp;rdquo; meaning the service only launches if the system is domain joined,
which is pretty lame. We can correct this one easily enough with the following command:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;sc triggerinfo w32time start/networkon stop/networkoff
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This will change the startup parameters of the service to launch whenever the network comes up and
stop whenever the network goes down, pretty much exactly what we want. Very nearly. My primary
impetus for the work was a slew of systems with faulty BIOS batteries. In my testing the &lt;code&gt;w32time&lt;/code&gt;
service would launch on boot but does not synchronize immediately. The service has two separate
settings &lt;code&gt;MaxPollInterval&lt;/code&gt; which defines the longest time between polls and &lt;code&gt;SpecialPollTimeRemaining&lt;/code&gt;
which is described in the documentation as:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;This entry is maintained by W32Time. It contains reserved data that is used by the Windows
operating system. It specifies the time in seconds before W32Time will resynchronize after the
computer has restarted. Any changes to this setting can cause unpredictable results. The default
value on both domain members and on stand-alone clients and servers is left blank.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;This means the system will perform its first poll eventually but we don&amp;rsquo;t know how long and we can&amp;rsquo;t
control when. This leaves me a little unsettled.&lt;/p&gt;
&lt;p&gt;When adding your NTP servers every piece of documentation I came across included an extra field in
the server name, i.e. &lt;code&gt;time.microsoft.com,0x1&lt;/code&gt;. When we add the &lt;code&gt;0x1&lt;/code&gt; we are telling W32Time to use
the registry key &lt;code&gt;SpecialPollInterval&lt;/code&gt; instead of the automagic &lt;code&gt;MinPollInterval&lt;/code&gt; and
&lt;code&gt;MaxPollInterval&lt;/code&gt;. The downside is that the default, for non-domain joined systems, is 604,800
seconds (7 days). Personally I&amp;rsquo;m a bit unhappy about that since drifts of even as little
as 5 minutes can cause problem. I could change this to 5 minutes, so it would resync relatively
quickly after boot, but that seems a bit much. I&amp;rsquo;m quite a lot more comfortable with 1 hour.&lt;/p&gt;
&lt;p&gt;This still doesn&amp;rsquo;t solve my problem of fixing the time &lt;em&gt;at&lt;/em&gt; boot. According to my research &lt;code&gt;w32time&lt;/code&gt;
has no notion of immediately sync on launch so the recommendations are to either join the systems
do a domain or to set a scheduled task to sync the time. Given the constraints of the systems
involved this leaves us with no choice but to use a scheduled task.&lt;/p&gt;
&lt;p&gt;The schedule tasked command line interface &lt;code&gt;schtasks&lt;/code&gt; actually gives quite a lot of ability to
customize the task. One of the standard triggers is &amp;ldquo;ONSTART&amp;rdquo; which sounds awesome because that will
run the task when the system boots up, whether someone is logged in or not. I like this but the NTP
service requires networking and I couldn&amp;rsquo;t figure out a way to set that as a delay condition, i.e.
&amp;ldquo;Run this task when the system boots unless networking isn&amp;rsquo;t up in which case wait and run it then.&amp;rdquo;
Instead I used an EventLog trigger. Whenever any network interface comes up, with a network
connection, the provider &lt;code&gt;Network Profile&lt;/code&gt; generates eventID 10000 which is stored in
&lt;code&gt;Microsoft-Windows-NetworkProfile/Operational&lt;/code&gt;. We can use this entry as the trigger for our time
syncs. The downside is that if the network drops out it will resync when it comes back up. Considering
how minimal the network impact of NTP is I&amp;rsquo;m actually pretty ok with it.&lt;/p&gt;
&lt;p&gt;Taking all that together we can add the new task to the end of my batch file as:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;schtasks /create /ru &amp;quot;SYSTEM&amp;quot; /tn &amp;quot;Initial Time Synchronization&amp;quot; /tr &amp;quot;w32tm /resync&amp;quot; /sc onevent /ec &amp;quot;Microsoft-Windows-NetworkProfile/Operational&amp;quot; /mo &amp;quot;*[System[Provider[@Name=&#39;Microsoft-Windows-NetworkProfile&#39;] and (EventID=10000)]]&amp;quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I tested it by making a bunch of crazy changes to my time. Things like moving it forward 30 years or
backwards 5. In every trial the time was correctly set as soon as the network plumbed whether as
part of a normal boot or me manually disconnecting/reconnecting the interface.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>Fixing Time Synchronization in Windows</title>
      <link>https://secopsmonkey.com/post/2014-05-02-time-synchronization-in-windows/</link>
      <pubDate>Fri, 02 May 2014 12:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-05-02-time-synchronization-in-windows/</guid>
      
        <description>&lt;p&gt;Keeping correct time is understandably pretty important. Without correct time logs are unreliable,
for both troubleshooting and investigations, domain logins don&amp;rsquo;t work since Kerberos tickets have
time based expiration, SSL/TLS won&amp;rsquo;t work for the same reason. In short, invalid time is all
kinds of bad for numerous reasons and the failures may not be obviously due to time skew.&lt;/p&gt;
&lt;p&gt;At ${dayjob} we have a rather large number of appliances deployed across the country and one
production run ended up with a batch of bad batteries. While the systems are up and running NTP does
a good job of periodically asking the time server what the current time is and setting it back if
the time ends up being off. Once the OS stops handling the clock the hardware falls back to the
motherboard to keep time and with the faulty battery the time gets reset to some preset value,
sometimes the oldest time the BIOS can store but more frequently when the BIOS was originally
compiled. Whatever the time is it will most likely be a long time ago, frequently years.&lt;/p&gt;
&lt;p&gt;When the device boots up it will ask the time server for the correct time&amp;hellip;problem solved right? As
Patience would say, there&amp;rsquo;s a kind of hitch. NTP client services get a little cranky if the time from
the NTP server is too different from their own and refuse to change the time. In the case of ntpd
on Linux will log a message and exit if the difference is more than 1000 seconds (16 minutes 40
seconds), Windows W32Time will keep running but not change the time if it differs by more than
54,000 seconds (15 hours). Most Linuxes account for a bad hardware clock by allowing you to perform
a one-time hard set of the time on boot, this brings the clock into shape such that ntpd can do it&amp;rsquo;s
job.&lt;/p&gt;
&lt;p&gt;Unfortunately for us our appliances run Windows which doesn&amp;rsquo;t have such a feature. After some
internal discussions the best option we had was a startup task to hard set the time which we all
felt was pretty terrible. Some careful and exceptionally boring reading of the &lt;a href=&#34;http://technet.microsoft.com/en-us/library/cc773263(WS.10).aspx&#34;&gt;Windows Time Service
Tools and Settings&lt;/a&gt; TechNet article solved the problem for me. There are two registry keys,&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;MaxNegPhaseCorrection
Registry path
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config
Version
Windows XP, Windows Vista, Windows 7, Windows Server 2003, Windows Server 2003 R2, Windows Server 2008, and Windows Server 2008 R2

This entry specifies the largest negative time correction in seconds that the service 
makes. If the service determines that a change larger than this is required, it logs 
an event instead. Special case: 0xFFFFFFFF means always make time correction. The 
default value for domain members is 0xFFFFFFFF. The default value for stand-alone 
clients and servers is 54,000 (15 hrs).
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;and&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;MaxPosPhaseCorrection
Registry path
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config
Version
Windows XP, Windows Vista, Windows 7, Windows Server 2003, Windows Server 2003 R2, Windows Server 2008, and Windows Server 2008 R2.

This entry specifies the largest positive time correction in seconds that the service 
makes. If the service determines that a change larger than this is required, it logs 
an event instead. Special case: 0xFFFFFFFF means always make time correction. The 
default value for domain members is 0xFFFFFFFF. The default value for stand-alone 
clients and servers is 54,000 (15 hrs).
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;By setting both keys to 0xFFFFFFFF the W32Time service will &lt;strong&gt;always&lt;/strong&gt; correct the time on every
time sync. It doesn&amp;rsquo;t fix the root problem of faulty hardware but it does keep affected systems
running while we go through the repair process.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>vSphere Client Display Trials and Tribble-ations</title>
      <link>https://secopsmonkey.com/post/2014-04-30-vsphere_client_display_trials_and_tribble-ations/</link>
      <pubDate>Wed, 30 Apr 2014 12:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-04-30-vsphere_client_display_trials_and_tribble-ations/</guid>
      
        <description>&lt;p&gt;I&amp;rsquo;ve had brief dalliances with both Windows and Linux as working environments but for the &lt;em&gt;vast&lt;/em&gt;
majority of my professional career I&amp;rsquo;ve used OSX as my desktop and laptop environment. Since I
almost exclusively deal with web interfaces and Linux systems&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a href=&#34;#fn:1&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;. There are a few silly things
that only work on Windows, like the vSphere client, so for those I keep a Windows 7 VM available.
The issue is that I pretty well entirely work off a laptop these days. Battery life on these things
is pretty awesome but not infinite. More importantly if I have this thing in my lap, and CPU usage
goes high or the discrete graphics card kicks in, the metal housing gets a little&amp;hellip;toasty.&lt;/p&gt;
&lt;p&gt;To fix both problems I opened up the display Performance Options
( Start -&amp;gt; Run -&amp;gt; systempropertiesperformance ) and setting the &amp;ldquo;Adjust for Best Performance&amp;rdquo;
options this unchecks all of those boxes. This basically disables every wiz bang piece of Aero and
gives you a reasonably thin (graphics wise) footprint. Everything&amp;rsquo;s awesome until I launch the
vSphere client and see this.&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-04-30-vsphere_client_display_trials_and_tribble-ations/fugly_vsphere.jpg&#34;
    alt=&#34;Mess Up vSphere Login&#34;&gt;
&lt;/figure&gt;

&lt;p&gt;There are also similar display problems once I actually get into the application itself. After
some trial and error I figured out that there&amp;rsquo;s only two display settings that get rid of the fugly
behavior.&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-04-30-vsphere_client_display_trials_and_tribble-ations/defugly_vsphere_effects.jpg&#34;
    alt=&#34;Make vSphere Less Fugly&#34;&gt;
&lt;/figure&gt;

&lt;p&gt;Checking those boxes and hitting &amp;lsquo;Apply&amp;rsquo; fixes us right up.&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-04-30-vsphere_client_display_trials_and_tribble-ations/not_fugly_vsphere.jpg&#34;
    alt=&#34;Not Fugly vSphere&#34;&gt;
&lt;/figure&gt;

&lt;p&gt;See? That&amp;rsquo;s what vSphere should look like. Unfortunately it means I still have to turn on compositing,
which was the biggest setting I wanted to shut down, but it&amp;rsquo;s at least better than nothing.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;This has been changing somewhat in my new role since we don&amp;rsquo;t have dedicated Windows and
Linux security guys but it still holds somewhat.&amp;#160;&lt;a href=&#34;#fnref:1&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>RADIUS for ASA on Windows Server 2012r2</title>
      <link>https://secopsmonkey.com/post/2014-04-25-radius-for-asa-on-windows-server-2012r2/</link>
      <pubDate>Fri, 25 Apr 2014 12:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-04-25-radius-for-asa-on-windows-server-2012r2/</guid>
      
        <description>&lt;p&gt;As old as it is RADIUS is still a pretty nice tool for getting non-Windows services to authenticate
against Active Directory. It&amp;rsquo;s pretty natively supported in most all network devices, has well
tested PAM modules, and is well understood by infrastructure systems like load balancers. Hell, it&amp;rsquo;s
even the preferred authentication type for some two-factor systems such as Windows Azure Multi-Factor
Authentication Server (né PhoneFacter). This long history and strong support makes it a nice
intermediary even if you are &lt;em&gt;not&lt;/em&gt; using some of the policy based access and accounting mechanisms.&lt;/p&gt;
&lt;p&gt;Choosing a RADIUS server can be a bit of an interesting endeavor. There are a number of options, the
most well known seeming to be FreeRADIUS, SteelBelted RADIUS, and Windows Network Policy and Access
Services. Since my use case is primarily a Windows shop the answer was pretty simple.&lt;/p&gt;
&lt;p&gt;I&amp;rsquo;m going to be making a few assumptions here. The RADIUS server is a domain joined system, you are
using an account with (at least) Domain Admins permissions, and you simply want to restrict
authentication to a group of users and not do any kind of advanced policy enforcement.&lt;/p&gt;
&lt;h1 id=&#34;installation&#34;&gt;Installation&lt;/h1&gt;
&lt;p&gt;Let&amp;rsquo;s assume you&amp;rsquo;ve already set up a base Windows box, I&amp;rsquo;m using 2012r2, that&amp;rsquo;s been joined to your
target domain.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Get into your target system, via RDP or Console or whatever.&lt;/li&gt;
&lt;li&gt;From the main Server Manager Dashboard click &amp;lsquo;Add roles and features&amp;rsquo;.
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-04-25-radius-for-asa-on-windows-server-2012r2/radius_step_2_add_roles.jpg&#34;
    alt=&#34;Add Role&#34;&gt;
&lt;/figure&gt;
&lt;/li&gt;
&lt;li&gt;Click &amp;lsquo;Next&amp;rsquo; a bunch of times, 3 times for me, until you get to the &amp;ldquo;Select server roles&amp;rdquo; screen.&lt;/li&gt;
&lt;li&gt;Click the &amp;ldquo;Network Policy and Access Services&amp;rdquo; checkbox.
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-04-25-radius-for-asa-on-windows-server-2012r2/radius_step_2_add_roles.jpg&#34;
    alt=&#34;Add Role&#34;&gt;
&lt;/figure&gt;
&lt;/li&gt;
&lt;li&gt;On the pop-up make sure &amp;ldquo;Include management tools&amp;rdquo; checkbox is checked and click &amp;lsquo;Add Features&amp;rsquo;.&lt;/li&gt;
&lt;li&gt;Click &amp;lsquo;Next&amp;rsquo; until you get to the Confirmation page, 4 times for me.&lt;/li&gt;
&lt;li&gt;Click &amp;lsquo;Install&amp;rsquo;.
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-04-25-radius-for-asa-on-windows-server-2012r2/radius_step_7_add_roles_install.jpg&#34;
    alt=&#34;Role Installation&#34;&gt;
&lt;/figure&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;You&amp;rsquo;ll see a progress meter that will somewhat quickly fill. Completion of the installation is
non-obvious but if you hover your cursor over the bar you will see the percentage and the message
will subtly change when it has completed. Once it&amp;rsquo;s finished click &amp;lsquo;Close&amp;rsquo;.&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-04-25-radius-for-asa-on-windows-server-2012r2/radius_step_8_roles_finished.jpg&#34;
    alt=&#34;Role Installation Success&#34;&gt;
&lt;/figure&gt;

&lt;h1 id=&#34;configuration&#34;&gt;Configuration&lt;/h1&gt;
&lt;h3 id=&#34;user-setup&#34;&gt;User Setup&lt;/h3&gt;
&lt;p&gt;Later on we&amp;rsquo;re going to restrict authentication to group membership. So before we go too far I want
to set up a group. If you have a better way to create security groups do so, I&amp;rsquo;m going to assume the
lowest&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Remote into your Domain Controller, or AD management system.&lt;/li&gt;
&lt;li&gt;Launch &amp;lsquo;Active Directory Users and Computers&amp;rsquo;.&lt;/li&gt;
&lt;li&gt;Browse to the OU that contains your groups.&lt;/li&gt;
&lt;li&gt;Select &amp;lsquo;Action&amp;rsquo; -&amp;gt; &amp;lsquo;New&amp;rsquo; -&amp;gt; &amp;lsquo;Group&amp;rsquo;.&lt;/li&gt;
&lt;li&gt;In &amp;ldquo;Group name:&amp;rdquo; type &amp;lsquo;VPN Users&amp;rsquo;. Click &amp;lsquo;OK&amp;rsquo;.&lt;/li&gt;
&lt;li&gt;Find &amp;ldquo;VPN Users&amp;rdquo; in the list, right-click and select &amp;lsquo;Properties&amp;rsquo;.&lt;/li&gt;
&lt;li&gt;Add an appropriate description.&lt;/li&gt;
&lt;li&gt;Select the &amp;lsquo;Members&amp;rsquo; tab and add users as normal.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&#34;permissions-registrations&#34;&gt;Permissions Registrations&lt;/h3&gt;
&lt;p&gt;By default a normal Computer object does not have sufficient permissions to view all of the
attributes for users. Specifically it needs access to the credentials and the dial-in properties.
Microsoft has a built-in mechanism for enabling this access that they call &amp;ldquo;registering&amp;rdquo;. There are
a number of ways of registering, however the only one that worked for me was using the command line.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Click the &amp;lsquo;Start&amp;rsquo; button.&lt;/li&gt;
&lt;li&gt;Type &lt;code&gt;cmd&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Right-click on &amp;ldquo;Command Prompt&amp;rdquo; and select &amp;ldquo;Run as administrator&amp;rdquo;. Click &amp;lsquo;Yes&amp;rsquo; for the UAC
elevation confirmation.&lt;/li&gt;
&lt;li&gt;When the shell comes up type: &lt;code&gt;netsh nps add registeredserver&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&#34;client-setups&#34;&gt;Client Setups&lt;/h3&gt;
&lt;p&gt;In order to accept RADIUS connections from an end device we have to configure it in the server as a
&amp;lsquo;Client&amp;rsquo;.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Click the &amp;lsquo;Start&amp;rsquo; button.&lt;/li&gt;
&lt;li&gt;Type &lt;code&gt;nps.msc&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;On the left hand sidebar expand &amp;lsquo;RADIUS Clients and Servers&amp;rsquo;.&lt;/li&gt;
&lt;li&gt;Right-click &amp;lsquo;RADIUS Clients&amp;rsquo; and select &amp;ldquo;New&amp;rdquo;.&lt;/li&gt;
&lt;li&gt;Enter the Display Name and IP address of the device that will be authenticating against your
RADIUS server.&lt;/li&gt;
&lt;li&gt;Select a shared secret.&lt;sup id=&#34;fnref:1&#34;&gt;&lt;a href=&#34;#fn:1&#34; class=&#34;footnote-ref&#34; role=&#34;doc-noteref&#34;&gt;1&lt;/a&gt;&lt;/sup&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-04-25-radius-for-asa-on-windows-server-2012r2/radius_client_step_6_new_client.jpg&#34;
    alt=&#34;Create New Client&#34;&gt;
&lt;/figure&gt;
&lt;/li&gt;
&lt;li&gt;Click &amp;lsquo;OK&amp;rsquo;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Now that we&amp;rsquo;ve defined our client the device is now able to actually talk to RADIUS and perform
authentication. However, before users can authenticate, we also have to create a policy to associate
with users.&lt;/p&gt;
&lt;h3 id=&#34;user-policy&#34;&gt;User Policy&lt;/h3&gt;
&lt;ol&gt;
&lt;li&gt;On the left hand sidebar expand &amp;lsquo;Policies&amp;rsquo;.&lt;/li&gt;
&lt;li&gt;Right-click &amp;lsquo;Network Policies&amp;rsquo; and select &amp;ldquo;New&amp;rdquo;.&lt;/li&gt;
&lt;li&gt;Type in a name for this connection policy and click &amp;lsquo;Next&amp;rsquo;.
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-04-25-radius-for-asa-on-windows-server-2012r2/radius_policy_step_3_name.jpg&#34;
    alt=&#34;Create New Policy&#34;&gt;
&lt;/figure&gt;
&lt;/li&gt;
&lt;li&gt;In &amp;lsquo;Conditions&amp;rsquo; click &amp;lsquo;Add&amp;hellip;&amp;rsquo;&lt;/li&gt;
&lt;li&gt;Select the &amp;ldquo;User Groups&amp;rdquo; option and click &amp;lsquo;Add&amp;hellip;&amp;rsquo;.
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-04-25-radius-for-asa-on-windows-server-2012r2/radius_policy_step_5_condition_type.jpg&#34;
    alt=&#34;Policy Condition&#34;&gt;
&lt;/figure&gt;
&lt;/li&gt;
&lt;li&gt;In the &amp;lsquo;User Groups&amp;rsquo; popup click &amp;lsquo;Add Group&amp;rsquo; and enter the &amp;ldquo;VPN Users&amp;rdquo; group we created earlier.&lt;/li&gt;
&lt;li&gt;Click &amp;lsquo;OK&amp;rsquo;, &amp;lsquo;Next&amp;rsquo;. Ensure &amp;ldquo;Access granted&amp;rdquo; is selected and click &amp;lsquo;Next&amp;rsquo;.&lt;/li&gt;
&lt;li&gt;The default options should be good. Make sure &amp;ldquo;MS-CHAPv2&amp;rdquo; and &amp;ldquo;MS-CHAP&amp;rdquo; are selected and click
&amp;lsquo;Next&amp;rsquo;.
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-04-25-radius-for-asa-on-windows-server-2012r2/radius_policy_step_8_authentication_methods.jpg&#34;
    alt=&#34;Authentication Methods&#34;&gt;
&lt;/figure&gt;
&lt;/li&gt;
&lt;li&gt;No constraints are necessarily necessary. Click &amp;lsquo;Next&amp;rsquo;.&lt;/li&gt;
&lt;li&gt;No specific policy settings are necessary. Click &amp;lsquo;Next&amp;rsquo;.&lt;/li&gt;
&lt;li&gt;Review the final options and click &amp;lsquo;Finish&amp;rsquo;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;At this point you should be able to edit your client device and add the Windows system as a RADIUS
server for authentication. That configuration will depend somewhat heavily on the client device is
may, or may not, be a post for another day.&lt;/p&gt;
&lt;div class=&#34;footnotes&#34; role=&#34;doc-endnotes&#34;&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id=&#34;fn:1&#34;&gt;
&lt;p&gt;You want this long enough to be awesome but be careful of the length because some devices have
issues with overly long secrets. If in doubt pick something about 30 characters long and be prepared
to change it if the device doesn&amp;rsquo;t connect.&amp;#160;&lt;a href=&#34;#fnref:1&#34; class=&#34;footnote-backref&#34; role=&#34;doc-backlink&#34;&gt;&amp;#x21a9;&amp;#xfe0e;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</description>
      
    </item>
    
    <item>
      <title>Introducing Infosec Happenings</title>
      <link>https://secopsmonkey.com/post/2014-04-24-introducing-infosechappenings/</link>
      <pubDate>Thu, 24 Apr 2014 12:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-04-24-introducing-infosechappenings/</guid>
      
        <description>&lt;p&gt;Way back in 2013 I was potentially going to be tagging along with my wife as she attended a work
conference in Las Vegas. I started looking around the Interweb for security related events happening
there in early January to give me something to do. After an hour I was getting somewhat cranky with
exactly how totally distributed the information was. Thus InfoSec Happenings was born.&lt;/p&gt;
&lt;figure&gt;&lt;img src=&#34;https://secopsmonkey.com/post/2014-04-24-introducing-infosechappenings/packscott_twitter_post.png&#34;
    alt=&#34;InfoSec Happenings Announcement&#34;&gt;
&lt;/figure&gt;

&lt;p&gt;I&amp;rsquo;m now excited to announce &lt;strong&gt;&lt;a href=&#34;http://secopsmonkey.com/infosechappenings/&#34;&gt;InfoSec Happenings v1.5: The Happeningier&lt;/a&gt;&lt;/strong&gt;. Instead of being just a
lame old shared Google Calendar with the &lt;code&gt;ics&lt;/code&gt; subscription link it&amp;rsquo;s now a lame old shared Google
Calendar embedded into a web page that &lt;em&gt;still&lt;/em&gt; contains a &lt;code&gt;ics&lt;/code&gt; subscription link.&lt;/p&gt;
&lt;p&gt;As always the calendar is a work in progress, I have some thoughts on a proper v2.0 but would love
to hear what you all would like to see. Leave me some feedback in the comments below or tweet me.&lt;/p&gt;
</description>
      
    </item>
    
    <item>
      <title>Cobbler on Ubuntu</title>
      <link>https://secopsmonkey.com/post/2014-04-18-cobbler-on-ubuntu/</link>
      <pubDate>Fri, 18 Apr 2014 12:00:00 +0000</pubDate>
      
      <guid>https://secopsmonkey.com/post/2014-04-18-cobbler-on-ubuntu/</guid>
      
        <description>&lt;p&gt;#Selection and Installation
Coming into a new environment is always interesting. One of the first tasks that came up coming into
this one was to throw up a provisioning server. Since we&amp;rsquo;re already standardized on Ubuntu I didn&amp;rsquo;t
want to immediately jump to Cobbler as the tool of choice. Imagine my surprise when, after
reviewing the options, I realized that a tool designed for Redhat is actually the best choice anyway.&lt;/p&gt;
&lt;p&gt;Luckily for us Ubuntu 13.10 upgraded to Cobbler 2.4, so when 14.04 LTS is released the upgrade path
should be easier. The list of packages is pretty easy:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;cobbler&lt;/li&gt;
&lt;li&gt;cobbler-common&lt;/li&gt;
&lt;li&gt;cobbler-web&lt;/li&gt;
&lt;li&gt;python-cobbler&lt;/li&gt;
&lt;li&gt;debmirror&lt;/li&gt;
&lt;li&gt;mkisofs&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;As you might expect, installation is as easy as:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;aptitude install cobbler cobbler-web debmirror mkisofs
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Cobbler is the service itself and cobbler-web is the web interface to the tool. Anything you can do
from the web can also be done from the command line, however, personally I find the web interface a
bit less cumbersome. The last two packages we&amp;rsquo;ll discuss later.&lt;/p&gt;
&lt;p&gt;Once the packages install everything should be good to go. It will pull in a fair number of
dependencies. Included in the list is apache which will also be minimally configured and started for
you. By default it only runs over non-SSL. If you want encryption, and you probably do, that will
have to be set up separately.&lt;/p&gt;
&lt;p&gt;Before you go to the web interface we&amp;rsquo;ll need to tell it how you want to authenticate. It supports a
number of options, including none. For our purposes we&amp;rsquo;ll just use system accounts. In a corporate
environment you&amp;rsquo;ll probably want to configure LDAP, against Active Directory. For our purposes we&amp;rsquo;ll
just use local system accounts. So we need to edit the file &lt;code&gt;/etc/cobbler/modules.conf&lt;/code&gt; and change
the authentication type. Find the appropriate section (line 23 on my system) and make it look like
the following.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;[authentication]
module = authn_pam
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now restart cobbler so it will see what we changed, service cobbler restart. At this point it&amp;rsquo;s
probably worth noting that the &amp;ldquo;authorization&amp;rdquo; section also has some useful bits in it. Here we can
also configure actual user level access to certain things. It can be a bit of a complicated
component and will require research of your own to enact.&lt;/p&gt;
&lt;p&gt;#Adding a Distro
Before we actually go to it, however, we need to add a distribution. To explain we need to go into
a little bit about how Cobbler arranges it&amp;rsquo;s internal objects. In truth it&amp;rsquo;s a fairly simple, and
short, hierarchy. At the lowest level we have ‘Distros&amp;rsquo;. This is a combination of OS and
architecture, so Ubuntu 13.10 on amd64. Next we have ‘Profiles&amp;rsquo;. A profile would be a general
configuration of a Distro. That is, we might have a profile called ‘ubuntu13.10-x86_64 Prod Sever
and another ‘ubuntu13.10-x86_64 Developer Server. Next we have systems. A system would be a specific
install instance and uses a single profile. So we would create a system called ‘www.example.com&amp;rsquo;
that uses the ‘ubuntu13.10-x86_64 Server&amp;rsquo; profile, has a single network interface called ‘eth0&amp;rsquo;
with MAC address 00:11:22:33:44:55, etc.&lt;/p&gt;
&lt;p&gt;For our purposes we really only care about the 64-bit Server instance of 13.10. So we&amp;rsquo;ll insert the
install CD, or mount the install iso, and run the command:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cobbler import --name=ubuntu13.10 --arch=x86_64 --path=/media/cdrom
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Cobbler will read the install media and generate a distro and profile for you. It also creates a
‘Repo&amp;rsquo; for that distribution as well. This ‘Repo&amp;rsquo; will be a locally hosted version of the software
tree. When you&amp;rsquo;re to the point of doing automated installations having a local mirror is pretty
handy. It saves you some of your upstream bandwidth, since you can download all the packages to your
provisioning server and then everything installs against it. It also allows you to be a bit more
careful about your software versions, you synchronize your tree today and freeze it. No matter if
you install a system today, or 3 months from now, it will be consistent.&lt;/p&gt;
&lt;p&gt;#Local Mirror Setup
What it won&amp;rsquo;t do is actually download all the packages. This is good for us because, by default, the
distro contains gobs more packages than we actually want. Ubuntu distributes packages under three
names:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;main - Officially supported packages&lt;/li&gt;
&lt;li&gt;universe - Boatloads of other software packages that aren&amp;rsquo;t supported&lt;/li&gt;
&lt;li&gt;multiverse - Somehow even &lt;em&gt;less&lt;/em&gt; supported and may not even have a &amp;ldquo;free license&amp;rdquo;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The above import wants to pull in ‘main&amp;rsquo; and ‘universe&amp;rsquo;, and the categories called saucy,
‘saucy-updates&amp;rsquo;, and ‘saucy-security&amp;rsquo;. The ‘security&amp;rsquo; category are all software updates that are
labeled as security fixes and ‘updates&amp;rsquo; is the combined list of all updates. This is where some
configuration time questions come into place. How much space do you have and how much do you want to
store locally? If you took the defaults you&amp;rsquo;ll be storing quite a lot of data, certainly more than
the 80GB my system had available.&lt;/p&gt;
&lt;p&gt;For our purposes we only wanted to pull in main RTM.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Log into the web interface (finally)&lt;/li&gt;
&lt;li&gt;On the left hand side under Configuration click on &amp;ldquo;Repos&amp;rdquo;&lt;/li&gt;
&lt;li&gt;edit ubuntu13.10-x86_64, click on &amp;ldquo;Advanced&amp;rdquo;&lt;/li&gt;
&lt;li&gt;Set &amp;ldquo;Apt Components (apt only)&amp;rdquo; to be main only.&lt;/li&gt;
&lt;li&gt;Set &amp;ldquo;Apt Dist Names (apt only)&amp;rdquo; to be saucy only.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Now that we&amp;rsquo;ve limited ourselves down a bit drop back down to the command line and run:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cobbler reposync
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This tells cobbler to do a few things in the background. First it will use the application
&lt;code&gt;debmirror&lt;/code&gt; to download all the packages in the main channel for 13.10. Then it will turn that into
a local repository that &lt;code&gt;apt-get&lt;/code&gt; can use. All told the local repository will be about 10GB, so
expect this to take a while. I ate lunch and chatted with the fellow in the next cube.&lt;/p&gt;
&lt;p&gt;Assuming nothing exploded and threw nasty error messages yet yell hooray because we&amp;rsquo;re ready to
start actually provisioning systems.&lt;/p&gt;
&lt;p&gt;#Creating Systems
Go back to the web interface and under ‘Configuration&amp;rsquo; select ‘Systems&amp;rsquo;. Click the
‘Create New System&amp;rsquo; button and let&amp;rsquo;s define our box. In the ‘General&amp;rsquo; tab we need to give it a name
and set what kind of install this is. The ‘Name&amp;rsquo; is entirely for your benefit and is only a label
that cobbler uses, so let&amp;rsquo;s call it &amp;ldquo;My First Provision&amp;rdquo;. Change the profile from ‘None&amp;rsquo; to
‘ubuntu13.10-x86_64&#39;.&lt;/p&gt;
&lt;p&gt;Flip over to the ‘Networking&amp;rsquo; tab. Type ‘eth0&amp;rsquo; into the ‘Add Interface&amp;rsquo; box and click ‘Add&amp;rsquo;. Now
make sure ‘eth0&amp;rsquo; is selected in the ‘Edit Interface&amp;rsquo; drop-down and insert the MAC address into the
‘MAC Address&amp;rsquo; field. If this will be a DHCP assigned interface you can leave the rest alone,
otherwise click the ‘Static&amp;rsquo; checkbox and fill out the page. Once done click the ‘Save&amp;rsquo; button.&lt;/p&gt;
&lt;p&gt;#Fixing The Boot Disc Environment
Next up is to build the bootable CD that we&amp;rsquo;ll use to kick off the installer. Before we can actually
create the CD we need to do a couple of steps. Drop back down to the command line again and run:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cobbler get-loaders
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This tells cobbler to go out to the Internet and download a bunch of files necessary to create a
bootable CD. There&amp;rsquo;s a bit of a hitch, though. If you pay close attention to the output you&amp;rsquo;ll see
this line come through.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;downloading http://www.cobblerd.org/loaders/menu.c32-3.61 to /var/lib/cobbler/loaders/menu.c32
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The file &lt;code&gt;menu.c32&lt;/code&gt; belongs to a program called syslinux which is what we&amp;rsquo;ll later use to create the
bootable install CD. The &lt;code&gt;get-loaders&lt;/code&gt; command downloads the &lt;code&gt;menu.c32&lt;/code&gt; file from syslinux version
3.61 but this version of Ubuntu uses version 4.05. Since syslinux is very dependent on consistent
versioning we&amp;rsquo;ll need to replace that file with the right one. Cobbler doesn&amp;rsquo;t kill the original
version so replacing it is as easy as copying over the right file.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cp /usr/lib/syslinux/menu.c32 /var/lib/cobbler/loaders/menu.c32
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;#Provisioning
Now we&amp;rsquo;re ready to make ourselves a boot disc. Run the command:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cobbler buildiso
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;A few seconds later you should have a file called &lt;code&gt;generated.iso&lt;/code&gt; in your directory.
Congratulations! This is what you&amp;rsquo;ll use to actually install your new server. Copy the file off your
Cobbler server and burn it to disc. Or, if you&amp;rsquo;re installing into a virtual machine, mount it on the
guest.&lt;/p&gt;
&lt;p&gt;Once you boot from CD you&amp;rsquo;ll be presented with a nice curses menu that should include all the
profiles and systems you&amp;rsquo;ve created. Select &amp;ldquo;My First Provision&amp;rdquo; and press enter. You should be
treated with a series of progress screens displaying what step of the install is happening. Since we
never told Cobbler what the root password should be, it&amp;rsquo;s really pretty ok since we give it a hash,
you&amp;rsquo;ll be prompted to enter the password. Except for that your system should entirely install by
itself and, once finished, reboot.&lt;/p&gt;
&lt;p&gt;At this point you should have a fully functional server installed entirely from your Cobbler server.
By default the configuration is pretty boring, for how boring see it yourself at
&lt;code&gt;http://your_ipaddr/cblr/svc/op/ks/profile/ubuntu13.10-x86_64&lt;/code&gt;. This is the default preseed file that
ships with the install.&lt;/p&gt;
&lt;p&gt;#Getting More Awesome
Now that we&amp;rsquo;ve gotten Cobbler set up and working I highly encourage you to look into some of the
more interesting features. First read up on preseed, since that will let you tailor the install to
your environment. Next look into Cobbler Snippets. These are little bits of code that will get
included into your preseed and let you do fun things like autodetect the size of the hard drive and
create different partitions. Or change the system&amp;rsquo;s timezone dependent on the DNS name. In my
opinion the templating is where the real power of Cobbler shows itself and is definitely worth your
time.&lt;/p&gt;
</description>
      
    </item>
    
  </channel>
</rss>
