In Splunk's new search language, there are several search operators that can help you. I'll describe only a subset of what is possible.

• 1) You can search for unexpected events by looking at those that do not cluster into large groups. For example, you can cluster the errors in the last hour and report on the events the belong in the smallest clusters (e.g., 'error | cluster showcount=true | sort - cluster_count | head 5′).
• 2) You can find unexpected events by finding values that are far from the standard deviation. For example, you can search for sendmail events with anomalous 'delay' values (e.g., 'sourcetype=sendmail_syslog | anomalousvalue delay action=filter pthreshold=0.02′).
• 3) You can use machine learning to find events that have unexpected values based on the past historical context (e.g., '* | anomalies blacklist=boringevents').
• 4) It's a little bit of a hand-wave - but you can do really cool graphical reports that often make anomalies visibly obvious. For example, you could create a timechart of average cpu_seconds by host, and visibly see problems (e.g., 'sourcetype=top | timechart avg(cpu_seconds) by host').
• 5) Finally, Splunk is expandable - you can define your own search operators. If you know how to find events interesting to you, you can write a simple script and trivially integrate it with the power of a search platform that deals for billions of events in seconds. Since Splunk uses a scalable map-reduce framework, your script will run in the map-reduce framework and scale automatically.

Once you have searches that find unexpected events, you can set alerts for them. You can also combine events together into 'transactions', and look for anomalies in groups of events.

This comes up every once in a while on the support channel (EFnet/#splunk), so I guess that means I should do a blog post on it.

If you're making changes to the authentication.conf file and want to reload Splunk's auth system without going through the web UI, you can use one of our internal functions to do it at the command line:

$ splunk _internal rpc-auth '<call name="syncAuth"><params/></call>'

This fires off the same call that the UI would use to reload the auth system, so it functions identically. Note that this is an authenticated call, so you'll need to use one of the standard authentication methods (-auth, splunk login, or the SPLUNK_USERNAME/SPLUNK_PASSWORD env vars...).

The answer is nearly always the same. "What are you currently running in your infrastructure? Do you have a log archive? What are you comfortable configuring?"

Most, if not all systems come with syslog built in. Setting Splunk up to handle syslog inputs is trivial. If you only deal with single line events then syslog is fine. You would just configure Splunk to use the Monitor input and point it to the target directory that you are storing your syslog log files in. Often this is /var/log or /var/adm depending on a Linux or Solaris installation.

If you have a medium scale deployment where you have lots of servers, you can configure syslog to listen to remote syslog hosts. Run Splunk on your receiver and you're done.

As an example, lets say we have a Linux deployment.

• Step one, configure syslog to "listen" to incoming messages. On most systems these days the syslog flags are configured in the /etc/sysconfig/syslog file. Append -r to the SYSLOGD_OPTIONS="-m 0 -r"
• On the sender hosts append to the end of the file "*.*                          @LOGHOST"
• Add an entry to your /etc/hosts file for the IP address of "LOGHOST"

Assuming your receiver has the /var/log directory set up create an inputs.conf in your $SPLUNK_HOME/etc/system/local/ directory with the following stanza.

[monitor:///var/log]
sourcetype = syslog
disabled = false
host = host_name


I like to recommend syslog-ng for both large scale deployments, and deployments where there is significant traffic. Syslog-ng allows you to use TCP rather than UDP to send your log messages. As we all know, UDP is lossy.. If you have too many messages for the network, interface, or host you are running syslog on you will drop data. Also, syslog-ng allows you to pre-filter messages upon their arrival into "buckets" to give you better control over your logs. Splunk can still be easily configured to monitor the target path and easily handle the naming of incoming systems, events, and dates.

To configure your Splunk host to properly get the hostname on a log archive with syslog-ng, you would have to make sure syslog-ng is creating the hostname in the path. For example, /var/log/archive/hosts/hostname/.../

The Splunk monitor stanza would look like this:

[monitor:///var/log/archive/hosts]
host_segment = 5
sourcetype = syslog

Where does Splunk Forwarders come into play here? (I knew you would ask)

Splunk forwards multi-line log events. This makes troubleshooting java apps, php apps, practically anything that uses this format, trivial. Typically I recommend using a mixture of inputs. I like using syslog/syslog-ng for collecting the log data to a central repository. This guarantees that you will always have the original data around. I then recommend configuring a Splunk instance to monitor the target directory of the syslog messages as well as pointing Splunk at the directories that contain the multi-line events. Best of both worlds.

What are the drawbacks of Forwarders? Just like conifiguring Splunk as a syslog receiver, if your splunk instance is down, you get no data.

So, often the best solution is to run Splunk Forwarders on those hosts that have multiline logs and use syslog/syslog-ng on your central server. Collect syslog with syslog-ng and collect app logs with Splunk. Best of both worlds.

http://www.splunk.com/doc/latest/user/UnsupportedCommands#inputcsv

We are talking about promoting it to public, so while it says unsupported it does work. Here's how:

I've got events from my webserver for my new domain and I want to see what real hits it's getting and not my own. They look like this:


66.249.70.86 - - [23/Oct/2008:01:42:21 -0700] "GET /category/admin/ HTTP/1.1″ 200 5158 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"


And I've gotten some traffic already:


$ ./splunk dispatch 'source=/var/log/apache2/mynewdomain_access_log | stats count'
count
-----
11424


It's a standard format that was automatically recognized as sourcetype access_common, so the extracted field "clientip" is already there. I create a csv file containing the values I want to exclude like this:


clientip
xxx.xxx.xxx.xxx
yyy.yyy.yyy.yyy
zzz.zzz.zzz.zzz


This file needs to exist relative to $SPLUNK_HOME/var/run/splunk, so to avoid specifying a path in my search I'll just put it there. Note that I could also have used xxx.xxx.xxx.* if I wanted to, wildcards are ok.

Now I can do this search:


./splunk dispatch 'source=/var/log/apache2/mynewdomain_access_log NOT [inputcsv mycsvfile.csv]'



$ ./splunk dispatch 'source=/var/log/apache2/myghettodatacenter_access_log NOT [inputcsv mycsvfile.csv] | stats count'
count
- -
121


and only get the ones that aren't from my network. This search also works from the UI as


source="/var/log/apache2/mynewdomain_access_log" NOT [inputcsv mycsvfile.csv]


splunkd log messages go in the file splunkd.log. (Note that if you move the existing file out of the way, a fresh one is created on startup if you want to work with only the messages from the current run.) They are controlled by the log.cfg file located in /opt/splunk/etc, which specifies the log level of messages by category:

rootCategory=WARN,A1
category.LicenseManager=INFO
category.TcpOutputProc=INFO
category.TcpInputProc=INFO
category.UDPInputProcessor=INFO


Messages can be set to, in order of severity: DEBUG, INFO, WARN, FATAL, CRIT. Setting a log level gets you messages at that level and higher, so default settings are typically INFO or WARN. When you change something in this file, you need to restart Splunk for it to take effect. When you restart with the - debug flag, it uses a similar file, log-debug.cfg, with a different set of settings for DEBUG messages. Not everything is set to DEBUG, because some of the categories are very chatty.

One of those is FileInputTracker, which even in log-debug.cfg is set to WARN. If you are having problems with data input from files, either indexing multiple times or not indexing at all, set this to DEBUG to get more about what is going on.

Now there is another way to enable and disable messages other than changing the file and restarting. If you want to permanently change settings, or you need to test a script that manages starting and stopping Splunk, you'll want to use these files. But you can also turn loglevels for categories off and on with a specially constructed search:

| oldsearch !++cmd++::logchange !++param1++::root !++param2++::DEBUG


This is the seach used for 3.3.x, for 3.2 and before remove the "| oldsearch" part. Yes, that is really the pipe, or vertical bar, character there. (And you will get the message "Search Execute failed because Setting root priority" when the search completes.) You can change any category to any loglevel with this, using the category name for the param1 value and the loglevel for param2. "root" is a special keyword for all messages, otherwise use the correct category name like "LicenseManager". log.cfg is not changed, and on restart you will revert to the configured settings.

One clever thing you can do with this is set up a scheduled saved search to turn on debugging only when you want it. If you have some problem that you know happens around midnight, you can set up one search to turn it on (set it to DEBUG) and off (return it back to WARN or INFO or whatever.)

splunkweb messages are controlled by a different mechanism, the SplunkWeb.tac file. If your problem is specifically with splunkweb, such as debugging LDAP settings in the UI, turn on these additional messages. You do need to restart splunkweb, but this can be done with "splunk restart splunkweb" rather than restarting splunkd along with it on a normal restart.

Change this line:

# set global logging level
appLoggingLevel = logging.INFO


To this:

# set global logging level
appLoggingLevel = logging.DEBUG


The additional messages are output in $SPLUNK_HOME/var/log/splunk/web_service.log file.

Hit the "play" button, sit back, and have a tour of the Splunk office. Click the button with 3 dots on it to jump to the next 3D space.

Buried deep in the index are a bunch of *.data files:

www.feorlen.org[feorlen]:/Applications/splunk/var/lib/splunk/defaultdb/db$ ls -lr *.data
-rw-r - r - 1 root admin 10276 Sep 3 07:41 Sources.data
-rw-r - r - 1 root admin 5085 Sep 3 07:41 SourceTypes.data
-rw-r - r - 1 root admin 252 Sep 3 07:41 Hosts.data
-rw-r - r - 1 root admin 21 Jul 26 19:19 EventTypes.data

You will find them in every bucket, they contain event counts for sources, sources, hosts and event types along with some timerange info. During indexing, these are constantly being updated. They are supposed to look something like this (note my timestamping oops there for host::grumpy):

$ more Hosts.data
0 0 2147483647 0 0
1 host::grumpy 11194556 900458000 1231448496 1220453014
2 host::www 1953184 1194131619 1220452994 1220452994
3 host::www.feorlen.org 2350 1207761050 1216665145 1216665145
4 host::localhost 7482 1203904810 1217973661 1217973661

Except when they look like this:

$ more Hosts.data
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@
Hosts.data (END)

That isn't very good. splunkd doesn't much like it when somebody messes with it's *.data files. There are also supposed to be at minimum Sources.data, SourceTypes.data, and Hosts.data. (EventTypes.data may legitimately not be there in some cases.) Your crash log will likely contain something like this:

Backtrace:
[0x00002B51C8EEFB6E] abort + 270 (/lib/libc.so.6)
[0x00002B51C8EE8266] __assert_fail + 246 (/lib/libc.so.6)
[0x000000000066661D] ? (splunkd)
[0x0000000000697BA6] _ZN23DatabasePartitionPolicy20getSourceWordForCodeEmmR3Str + 182 (splunkd)

and here is the real smoking gun in splunkd_stderr.log:

splunkd: /opt/splunk/p4/splunk/branches/3.2/src/pipeline/indexer/TimeInvertedIndex.cpp:974: void TimeInvertedIndex::getSourceWordForCode(long unsigned int, Str&): Assertion `_sourceMetaData != __null' failed.

Ok, so you've got a horked *.data file. Where? Well, based on frequency of writes, it's going to be in a db-hot directory because that is where active indexing is going on. And the most active indexes are usually fishbucket, _internal and defaultdb. Start by looking for *.data files that are binary. Here's one way you can find which files are binary, a big clue on where the problem is:

$ cd /opt/splunk/var/lib/splunk
$ find . -name *.data | xargs grep "." % | grep Binary
grep: %: No such file or directory
Binary file ./_internaldb/db/db-hot/Hosts.data matches
Binary file ./_internaldb/db/db-hot/Sources.data matches
Binary file ./_internaldb/db/db-hot/SourceTypes.data matches
Binary file ./fishbucket/db/db-hot/Sources.data matches

file will do it also, but beware false positives:

$ for i in `find . -name *.data`; do file $i | grep -v text ;done
./_internaldb/db/db-hot/Hosts.data: data
./_internaldb/db/db-hot/Sources.data: data
./_internaldb/db/db-hot/SourceTypes.data: data
./defaultdb/db/db_1214955936_1210836930_38/Hosts.data: Bio-Rad .PIC Image File 2352 x 12297, 14601 images in file

Another check is to see if the line numbers in the file are in ascending order. If they aren't, then something is seriously wrong:

for i in `find . -name *.data`; do sort -nc $i;done

Have a look at these files and see what's in them. If they are only partially corrupted, you may be able to edit out the garbage. If they are totally full of junk, you will need to find replacements. For _internaldb and fishbucket, you may not care if your event counts are exactly correct so you can lift some files from another bucket. If the problem were in defaultdb or another index containing your real indexed data, you'll need to pay more attention to the contents.

In the simple case, if the files in db-hot are trashed, see if there is a warm bucket next to it you can copy some from. Warm buckets are in the same directory as db-hot and look something like db_1218802821_1218658318_17. Copy the *.data files from there into db-hot and try to restart Splunk. If it does, then you are good to go. If not, that means there is more damage to repair. If there are other binary *.data files, make sure you deal with all of them.

This should handle the most common types of problems. I'll go into more detailed debugging and reconstruction in another post.

I'll show you how to write your own search command.

Suppose you want to make a new “shape” command in python that returns the shape of an event - tall, short, thin, wide, etc.  There are just three simple steps:

• Step 1) Tell splunk about this external command in  commands.conf...

[shape]
filename = shape.py

• Step 2) Authorize users to run this command in authorize.conf...

[capability::run_script_shape]
[role_User]
run_script_shape = enabled

• Step 3) Write the code!  Here is shape.py...

   import splunk.Intersplunk 

   def getShape(text):
        description = []
        linecount = text.count("\n") + 1
        if linecount > 10:
            description.append("tall")
        elif linecount > 1:
            description.append("short")
        avglinelen = len(text) / linecount
        if avglinelen > 500:
            description.append("very_wide")
        elif avglinelen > 200:
            description.append("wide")
        elif avglinelen < 80:
            description.append("thin")
        if text.find("\n ") >= 0 or text.find("\n\t") >= 0:
            description.append("indented")
        if len(description) == 0:
            return "normal"
        return "_".join(description)            

   # get the previous search results
   results,unused1,unused2 = splunk.Intersplunk.getOrganizedResults()
   # for each results, add a 'shape' attribute, calculated from the raw event text
   for result in results:
        result["shape"] = getShape(result["_raw"])
   # output results
   splunk.Intersplunk.outputResults(results)

It works!  Show me the top shapes among events with more than one line...

$ splunk search "linecount>1 | shape | top shape"
shape                count  percent
-------------------  -----  ---------
tall_indented           43  43.000000
short_indented          29  29.000000
tall_thin_indented      15  15.000000
short_thin_indented     10  10.000000
short_thin               3   3.000000

Just to review, here are the files we made...

apps/example/bin/shape.py
apps/example/default/authorize.conf
apps/example/default/commands.conf

Now go out there and make cool extensions to Splunk!

REALLY BIG WARNING

Don't do this to any Splunk instance you like. You will be unhappy later. Throw away your dummy instance when you are done so you don't confuse anybody.

Set up a new instance of an appropriate version, the same or more recent as the original and appropriate architecture (ppc/sparc or intel.) Get it all working with the correct ports so you don't conflict with anything else that may be running on the machine. Since it won't be indexing, the license doesn't matter. Start and then stop so the first run stuff is done.

Change some things so it won't touch the index:
./splunk clean all -f
rm /opt/splunk/bin/splunk_optimize
rm /opt/splunk/etc/system/default/inputs.conf (or wherever it is in your version)
edit /opt/splunk/etc/system/default/indexes.conf to comment out the line frozenTimePeriodInSecs = 2419200 in [_thefishbucket] stanza
If it's large, you'll want to also comment out maxDataSize = 10

rm -rf /opt/splunk/var/lib/splunk/fishbucket/*
copy the contents of the fishbucket index you have into the now empty directory (don't accidentally create an extra fishbucket/fishbucket directory!)
remove any archives or other temporary files you left lying around in the index directories

Start this instance and now you can search for index=_thefishbucket. It helps to exclude the Splunk internal files with something like this:

index=_thefishbucket NOT filename::/opt/splunk/var/log/splunk/license_audit.log NOT filename::/opt/splunk/var/log/splunk/metrics.log NOT filename::/opt/splunk/var/log/splunk/searchhistory.log NOT filename::/opt/splunk/var/log/splunk/splunkd.log NOT filename::/opt/splunk/var/log/splunk/splunklogger.log NOT filename::/opt/splunk/var/log/splunk/web_access.log NOT filename::/opt/splunk/var/log/splunk/web_service.log

Your full path may vary. What is left is all the files being monitored by the instance.

Also, if you are a UI engineer using Java or .NET or AJAX (jQuery, ExtJS, etc..) technologies, and are motivated to move to ActionScript/Flex, we will provide you with the tools and mentoring to be successful in this position.

Experience in building network topology visualizations is a big plus!

All resumes can be emailed to me directly.

48a304b3 initcrc::5f66db978a1ff3a3 seekcrc::bc96de428cc0b5e6 seekptr::414063 modtime::1218643123 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log

The fields are:

timestamp (epoch time, in hex)
CRC of the first 256 bytes of the file
CRC of the 256 bytes where we were last reading
seek pointer for where we are in the file
the time the file last changed
the full path to the file.
the full path to the source, which is usually the same as the file but could be the archive the file came from.

When the file monitor processor looks at a file, it searches the fishbucket to see if the CRC from the beginning of the file is already there. If not, the file is indexed as new, If yes, then we check the CRC of where we were reading against the saved value in seekcrc. If it matches and the file is longer than the saved seek pointer, then there is new stuff at the end to read. If the top of the file matches but the seekcrc doesn't, or the seek pointer is beyond the current end of the file, then something in the part we have already read has changed. Since we don't know what might have changed, we just index the whole thing. (You can control this: see CHECK_METHOD in props.conf.spec.)

If you want to track what is happening with a particular file, you can search for all the events in the fishbucket associated with it by the file or source name (like source::/var/log/apache2/feorlen_org_access_log.) If you check the seekptr and the modtime, they will only be increasing with time (note that events are returned most recent first, so this list is newest to oldest.)

48a3084d initcrc::5f66db978a1ff3a3 seekcrc::3e746e9f66897965 seekptr::414a40 modtime::1218644042 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a307d9 initcrc::5f66db978a1ff3a3 seekcrc::77f6d8313fc689ba seekptr::41419b modtime::1218643929 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a3062e initcrc::5f66db978a1ff3a3 seekcrc::2cc30b86b37c646 seekptr::4140fc modtime::1218643502 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a304b3 initcrc::5f66db978a1ff3a3 seekcrc::bc96de428cc0b5e6 seekptr::414063 modtime::1218643123 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a300d3 initcrc::5f66db978a1ff3a3 seekcrc::8db2f52ef6f75c91 seekptr::413fa4 modtime::1218642130 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2fc7a initcrc::5f66db978a1ff3a3 seekcrc::881375418e194bd5 seekptr::413f06 modtime::1218640999 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2f996 initcrc::5f66db978a1ff3a3 seekcrc::c596371ec4c573d4 seekptr::413e6c modtime::1218640260 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2f80c initcrc::5f66db978a1ff3a3 seekcrc::2e686cf0dd2f62bb seekptr::413dce modtime::1218639883 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2f25a initcrc::5f66db978a1ff3a3 seekcrc::b2e489862ed72c79 seekptr::413d1d modtime::1218638406 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2f1d1 initcrc::5f66db978a1ff3a3 seekcrc::58af0c6446e96bf5 seekptr::413c7f modtime::1218638289 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2f19d initcrc::5f66db978a1ff3a3 seekcrc::16fdb83b48965067 seekptr::413bbe modtime::1218638236 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2f05b initcrc::5f66db978a1ff3a3 seekcrc::fbb8700a35cfdfcb seekptr::413b25 modtime::1218637915 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2ebc5 initcrc::5f66db978a1ff3a3 seekcrc::ddbac21aa7386a6 seekptr::413abd modtime::1218636714 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log

Anything other than this indicates a big problem with the file, like it is getting re-indexed when it shouldn't. (Some files you do want to re-index when they change, but not normal logfiles that roll.)

So why do I care?

Every Splunk instance has a fishbucket index, except the lightest of hand-tuned lightweight forwarders, and if you index a lot of files it can get quite large. As any other index, you can change the retention policy to control the size via indexes.conf. But since it tracks what files the instance has seen, you have to consider carefully before you change the retention policy. If you retire data from the fishbucket for files that still exist on the host, it will "forget" it saw them and next time around they will get re-indexed.

As always feel free to bug me if the app has any problems.
e.

**** UPDATE - 10/10/08 ****

Hey all,
I update the latest release - 1.7 - to fix a shutdown bug.
Turns out that in prior releases when spunk was shut down that the VMWare app kept running.
This release not will terminate the VMWare app when splunkd goes away.

If you would like to test or run without splunk you can pass in the arg.
java -jar splunk.jar - standalone

** see instructions below on how to run the above command **
As usual, drop me a line if you have any questions.
Good luck with 1.7

**** UPDATE - 09/16/08 ****

Thanks to more testing i have found and fixed a few critical bugs.
Updated APP version 1.6 >> here <<

• there was a static var preventing the multiple server configs from working. Should be fixed, and multiple servers in the vmware.conf should work.
• Ibm jvm's should work - ie AIX should now work
• Added new saved searches and a few dashboards ( thanks to raffy

As usual, please let me know if you find any bugs.
I'll type up some notes on my VMworld experince

Cheers,
e

**** UPDATE - 09/08/08 ****
Thanks to lots of folks trying it out i have found a critical bug that was preventing much of the data from getting indexed. This latest release 1.5 should have that fix and everyone should see all the wonderful VMWare data in the index.

As usual, bug me if it does not work or you have any questions.

If you have made changes to vmware/local/vmware.conf and not to the file in default you can just untar this version on top of your old one. If you are making changes to the default/vmware.conf file, i'd move that to local/vmware.conf that way when i ship updates it will not blow away your conf changes. We ship only default and not local/vmware.conf.

Thanks again to everyone that helped find bugs!

e.

**** UPDATE - 08/27/08 ****
I have updated the app with a few fixes found in the field.

• hopefully fixed issue on AIX (IBM jvm )
• added output of host/vm name on update messages. It was hard to tell where the messages were coming from
• added more debugging infor on startup to help debug connection issues.

Things that are still under-investigation.

• Pointing at lots of ESX servers and not VC. Seems as though some data is not coming back from ESX.
• Making work with older jvm's ( currently it seems i require 1.5)

**** Original Post 08/10/08 ****
I've wanted to release this a few months ago but the project keeps getting stuck on the back-burner. Finally I've cleaned it up and had a few people try it and it seems to work well. I'm sure there are configurations and versions out there that will have issues - please write me back ( my first name at splunk.com ) if it does not work as advertised.

Reading the below makes it sounds more difficult that it really is. Just download, un-zip, change the server url, username and password in the vmware.conf file, restart and go! This really is the first pubic release and i'd love to get more feedback. I'll more than gladly send you Splunk tee shirt of your choice if you help find bugs or have useful suggestions!

Why you want to give it a try:
This vmware app is a cool way to keep track of what your VC and ESX servers are up to, what instances are running where, when they are under load, when instances move, when they have errors, and much more. Since all the data is indexed in Splunk, it's easy and quick to search for problems and report on your virtual sprawl.

How it works:
This app will connect a splunk server to any number of Virtual Center and/or ESX servers and grab/index the events, logs, properties, performance data, and anything else I can get my grubby mitts on. It's easy to hookup and get going, so if you use Virtual Center or ESX than give this app a try. I'll explain how to install/setup, how to trouble shoot, and what you will see when you get it working. You will need to install splunk or use an existing Splunk server. See the configuration file for settings on how often to pull data. Also near the end of this post i give example searches to explain the data.

After installing you get cool graphs like this one showing CPU Usage by Guest by Time:

Add Inside-out monitoring
Its optional but if you can also put splunk on the guest OS's as light weight forwarders and you will get a brilliant inside out view where we capture not only what VC/ESX thinks but what the guests are seeing on the inside. My best practice is to put splunk on the guests and capture basic logs as well as OS performance metrics, what apps are running, how much mem/cpu they are taking, etc. You can get the Unix/Linux version here and the windows here. Of course its not required and you get a ton of value out of just with the basic vmware app's monitoring of VC/ESX.

INSTALLATION:

**Important**
This app requires a JVM be installed on the same box as the splunk server. I know this is less that optimal. Please bug your local VMWare rep and tell them to make me REST API's and not SOAP API's. The VMware API's are hideously over complicated - Please dear VMware make a simple REST interface.

1) Make sure java is present and set the JAVAHOME environment variable. If not already set you must be set JAVAHOME to the directory that contains the java binary.

2) To test the variable is set correctly, try and run the following on the command line
windows> "%JAVAHOME%\bin\java
linux/unix> $JAVAHOME/bin/java

If it worked it should spit back a bunch of options to pass to the java command. If its not set right you will get some kind of file not found error.

3) Grab the vmware.zip file HERE.

4) Unzip the file - and copy the resultant "vmware" directory to your SPLUNK_HOME/etc/apps/ directory. When done the following directory should exist: SPLUNK_HOME/etc/apps/vmware.

CONFIGURATION:
There are a few config settings to make the app work.

5) First you need to let Splunk know where your VC or ESC servers are. Edit the vmware/default/vmware.conf configuration file to point to your vc or esx servers. If using VC you need not specify all ESX servers under management, splunk will get the list from VC. The config file contains one or more of the following stanza's ( the unique_name can be anything you like so long as its unique):
[vmserver:unique_name]

For each [vmserver] stanza be sure to set:
url=https://your_server_IP/sdk
username= your_user
password=your_passowrd


Note that the url should be the ipaddr of your server with "/sdk" at the end - for example "url=https://10.1.1.35/sdk". A good way to test that the url and username/password are correct is test using a web browser. Take the url you have entered above and replace the "sdk" with "mob". Use the web browser to navigate to that url and make sure it asks for username and password and that the values you entered above will authenticate correctly. If the "mob" url works with the username and passowrd you entered than splunk should have no trouble.

With those three set you should be up and running after a restart.
The rest of the config file should be self explanatory and is included end of this post for reference but you should not need to change anything else.

Testing and Troubleshooting:

6) It's best to test running the vmware app outside of splunk first.
You'll need to make sure that SPLUNK_HOME is set for the test.

** On Windows **:
set SPLUNK_HOME=your splunk directory
#note it does not like it when i add quotes around this path - try with no quotes.

Then run the app by hand
> cd %SPLUNK_HOME%\etc\apps\vmware
> java -jar lib/splunk.jar

** On others ** :
export SPLUNK_HOME=your splunk directory

Then run the app by hand:
> cd $SPLUNK_HOME/etc/apps/vmware
> java -jar lib/splunk.jar

It should spit out all sorts of vmware data. If it throws an error its likely that SPLUNK_HOME or JAVAHOME are NOT set. Remember SPLUNK_HOME will be set by the server when the server runs the script. You need only set it for testing.

If it does not work, likely the exception will have something useful in it such as connection refused ( bad auth ) or a 404 error in which case the url is incorrect.

If you get any non-obvious errors email me ( my first name at splunk.com ).

7) Try running in splunk.
If the above test works than you should be able to just restart splunk and all should be good. The way to tell if its working is that you will get events with sourcetype vmware and vmware_api.

8 ) If you do NOT see events of type vmware_api on the dashboard than try the following search:
"index=_internal error"
and
"index=_internal splunk4vmi.py"

You should see some kind of error or warning that is hopefully obvious. If not again email me and i'll sort you out.

Using the App

At this point it should be working and you should be able to search for cool stuff.
Here is a quick overview of what splunk is indexing:

After restarting you should see a bunch of logs from vwmare and at least two new sourcetypes ; vmware and vmware_api. Below is a screen shot of my dashboard after restarting - notice the vmware logs and the vmware_api event counts.

The vmware sourcetype is for the actual vwmware logs while the vmware_api sourcetype is for the API calls. It can take a minute before they show up so if they are not there, try again after a minute. If you still do not have the logs that likely means the logs path in the vmware.conf if incorrect and you should make sure the path is correct or contact me.

If you do not see the API calls than there is likely an auth or url error that should have been caught when you did the manual test above. Try retesting by hand above - if the by-hand method works but not through splunk than contact me.

I've just started to explore the logs that come back - there is a ton of information in them but my test infrastructure is not all that insteresting so i'm not sure what goodness you all might find in them. Poke around the files and see what you see and bug me if you see anything interesting i can make them into alerts / reports.

The meat of the data is from the API where we pull everything we can.
Most useful are:

1) Metrics
Every few seconds we captures the metrics for all VM's, including

2) Events
I'm not sure the scope of these but it looks like interesting events kicked out by ESX. Someone with a larger VMware installation might find far more interesting events than i see on our infrastructure.

3) Updates:
It looks like when anything changes, we can an update.

4) Inventory:
I periodically just capture the inventory tree. It's more for debugging than perhaps useful in a production environment but it does not cost much to get and it can be useful.

Thanks to Christina we do ship with a bunch of saved searches. After installing you should see them, they all start with 'VM:'. They are named to be somewhat obvious, again let me know if they dont work or you have some better ones to add to the default app. Try some of the Metrics and Status saved searches to make sure your install is working.

• VM: Investigation CPU load on all guests sharing ESX server
• VM: Investigation Find ESX Host for Guest
• VM: Investigation Find Guests sharing ESX Server - Non FQDN
• VM: Investigation- Find other VMs sharing ESX Host
• VM: Investigation- Processes on hosts sharing ESX Server
• VM: Investigation- Running processes on other guests on same ESX server
• VM: Metrics- CPU by Guest last 60 minutes * VM: Metrics- Host Memory Usage last 15 minutes
• VM: Metrics- Host Memory Usage last 60 minutes
• VM: Metrics- Memory by Guest last 60 minutes
• VM: Status- Free Space by Datastore
• VM: Status- Running Guests
• VM: Status- Running VMs

That's about it.
Like i said, PLEASE email me if you have bugs or suggestions.
I'll plan on updating the app with whatever feedback i get from folks. So please, help me out and get yourself a tee shirt.

Kind Regards,
e.

P.S. - there is a sample of the config just so that you can see what's in it without downloading:
- - - - - - - - - - - - -
The following are the important values in the config file:


[vmserver:demo]
url=https://10.2.1.151/sdk ## This is the url to the vc or esx server
username=your_username ## user name to auth against the server. If you are not sure of its value point we browser at the above url and check the web auth, it will be the same.
password=your_passowrd ## we will support non-clear text in the near future.
ignorecert = t ## for now leave as true (t), we will soon support checking of certs
loggingLevel = error ## to turn on debugging values are [error, warn, info, debug ]

index_logs = t ## should we index logs (t)rue or (f)alse
logs_interval = 300 ## how often to get log changes...
logs_localpath = ../var/spool/vmware ## the logs are copied from vc/esx to the this directory where splunk will pick them up for indexing


Serveral customers have reported that DeploymentServer configuration bundles were not taking effect, only to realize after several troubleshooting cycles that there was some configuration in /etc/system/local that was preventing that from happening. Note that any configuration in /etc/system/local will always take precedence over any other configuration in the system - even deployed bundles.

So, if you are stuck in this position, please make sure to check your /etc/system/local before hitting the panic button!

I've got another post here in the occasional "Help Me Help You" series, this time I'm going to digging into case writing.

I was talking with the some of the engineers the other day around the bar about an issue that one of our field guys opened. One of the engineers mentioned a piece of information that totally changed the way the rest of us were going to handle the issue. This got us to talking about how some people write great cases and others don't. The ones who write good cases usually get their issues resolved first (often times closing the issue with the first response from a member of my team), the ones who write "bad" cases generally have a back and forth exchange.

That got me thinking that maybe I should take a sec to talk about what makes a good case. I'm going to try mapping out a basic template for submitting an issue. This is by no means limited to Splunk and is most definitely not a de facto standard. Rather it is a compilation of things that always make my life easier when my customers can provide them.

• Backstory: Like I mentioned in my previous post I don't work in the cube next to you, I don't see the same things you see, know the same things that you know.
  Often times I get cases with a description like "I came into work this morning and discovered that this thingy that was working yesterday isn't working today. What gives?" In digging into the issue the customer remembers that last night was the weekly maintenance window and one of the other guys was making some changes on the box and it is this change that caused things to go wonky.
  I guess what I am getting at here is that it helps to know what led up to the issue. Flushing out the supporting data points can be a big help in piecing the problem together. Even if you think it is unrelated include it, it can't hurt. The worst thing that can happen is you spent a few more bits and thankful bits don't cost what they used to. I've also found that when I take the time to think about _all_ of the things that led up to the event in question the light bulb over my head starts to flicker and maybe I can figure it out before enlisting someone else.
• Impact: Do you have to commit seppuku if this issue is not resolved in the next hour? If you do you may want to include that in the initial report, it will really help with prioritizing the issue. Are others unable to do their job because of this, we want to know. If you're asking a question for your own edification share that as well - helps us to prioritize other issues and formulate the best answer for you. Big fires often require an immediate fix and you don't really care about the inner workings of the fix just that it works. If you are trying to learn something you want the opposite.
• Priority: We all deal with fires (some bigger than others) let the guy on the other end know how you need the issue treated. Support folk inherently want to help (why else do we do this job? It isn't for the unlimited supplies of handi-snacks) and if you say I need this now we will make every effort to deliver.
• Data Samples: One of my new favorite shows is The First 48 which follows real homicide cops as they investigate murders. Each episode always starts off with the cops going to crime scene collecting every potential piece of evidence. They don't know what is relevant and what is not, so they assume it all is. The same is true when troubleshooting an issue with software. The more data points I have to work with the better position I am in to figure out what is going on.
  If splunk isn't parsing a field in a given file include a copy of said file along with your configs. If the UI is acting weird take a screen shot. If performance is an issue include the results of your tests to determine that things are slow along with the tool(s) used to produce the results.
• Repro steps: If you can trigger this issue on demand, please share. Knowing the exact path traveled will often make root cause analysis that much easier. Screen shots of each step are very helpful (a picture is worth more than a 1,00 words) in describing an issue.
• Your investigation: I find it is really helpful to know what you have done to try to figure out a problem. It saves time because I wont ask you to perform steps that you said you've done and you wont get frustrated at me for asking you to do work again. It also gives me insight into your investigative process - if you are thorough I am more inclined to trust your results at first glance. If you are vague or unclear I have to assume that the information you are providing is incomplete. This is not to say that what you are giving is bad/wrong/stupid, rather it is not the full story.

Ok I'm sure there is more that I can say here but this post is getting kind of long, my fingers are tired of typing, and I need to answer some cases.

Splunk doesn't support Safari officially yet and MobileSafari is a whole 'nother animal, but there are other things you can do. You can talk to the REST endpoints just fine. Here I have a Live Tail search running from the browser, talking to my production server.

Dashboards create username_* files in $SPLUNK_HOME/var/run/splunk to persist the dashboard data. There is also a directory for each username with *.csv files. Delete the username_* files (like "admin_KB indexed per hour last 24 hours") and the *.csv files and the next time you refresh the dashboard, it will reload.

This is not an elegant solution by any means, but it does work. While you could just delete the files for the search in question, there is no simple way to identify which csv file is associated with it. Just don't go messing with the other files in this directory, you will be Very Unhappy if you do.

Around here is synonomys with filing 34 bugs between sunday 9PM when we push bits to the site and 9AM when we get in to the office. I dont mean the usual the UI-is-off-by-10-pixels but complex indexing or distributed search bugs. Well, sometimes is its a trivial thing we missed, but usually he is usually pushing splunk to its limits. Its not often that a CTO and "industry expert" is the one to personally put splunk through its paces - but it's RJ is like that and gets his hands dirty - and splunk is the better for it.

RJ and Voxeo are one of a few, but quickly growing, number of companies that are using splunk in a multi-tenant environment. This means using splunk to to collect data across multiple tenants in a hosted environment and then using splunk for searching and reporting on a per customer basis. Often the output of the searches/reports is rendered for the customer do they can see what is going on within the service. Customer dashboards and activity reports are a common usecase for splunk. Below are some of the images from the voxeo service:

On the Voxeo blog there is a nice description and even a cool video introduction:

Lessons learned from these initial deployments are having a significant effect on our upcoming 4.0 release. First and foremost we will provide a much better html "module" system so that you can embed splunk modules in other webpages. Secondly, we will be having the overall splunk UI more configurable and modular so that multi-tenant customers can build even more custom UI's.

One other very interesting trend is using splunk for SaS using cloud services. Often these uses have some kind of multi-tenant .... It wont be long before splunk makes deploying in the cloud even easier. More in a post to come but do drop me aline if you want to use splunk in the cloud and i can give you some hints.

In the mean time if your looking for the best push-it-to-the-limits beta tester, contact RJ!
Thanks RJ

e.

I just wanted to congratulate Nathan over at FlowingData for crossing the 3100 subscriber mark.

FlowingData is a fantastic example of the hidden value in the data all around us. As more and more of what we do is documented by computers the impact of statistics has become less of a hard-core math geek sport and more within the reach of anyone's curiosity. His daily posts are a constant reminder of how statistics has become a crossover genre.

Thank you Nathan!
e

You can configure the widget from the Widgets page and it supports multiple instances with different configuration. Right now the actual search string is hardcoded because I'm doing some extra mangling to get the search terms the way I want anyway, but I'll be adding that to the configuration options also. Eventually there will be a way to cache results so you don't do the search each time the page is loaded.

Since there is still work to do to make it more generic, I haven't uploaded it to the WordPress site. But here is the basic PHP code to play around with. In fine programming tradition, I learned quite a lot by picking apart existing WordPress widgets, in this case Random Image and Twitter Tools. This widget requires the Splunk PHP SDK, by default my code is expecting it to be in the same directory (which is probably going to be something like wp/wp-content/plugins/widgetname.) There are a few things it depends on, you can find the details at the Google Code page.

You can find the widget here:
splunk_statsphp1

Note: updated version posted 31 July 08.

Here's a sample of the kinds of events I'm looking at. I have some extra field extractions because it's a custom format and not exactly access_combined, but I get the referer in there. What I want to display is the actual search string, in this case "drum+carder". I have to strip out the '+' between words because otherwise it doesn't wrap nicely in my narrow sidebar. (I'm sure I could fix this in my theme somehow but Eric Meyer I'm not.)

xxx.xxx.xxx.xxx [15/Jul/2008:12:08:07 -0700] "GET /tag/drum-carder/ HTTP/1.1&#8243; 200 "http://www.google.com/search?hl=en&#038;pwst=1&#038;q=drum+carder&#038;start=10&#038;sa=N"

You can go look at the code if you really want to know, but here are a few comments on what it's doing:

I only want a couple results, so to make the search as fast as possible I'm limiting what I get back.
// how many results to get?
$dispatchProps['max_count'] = 3;

Also there's no need to have the default time to live, so set the timeout to something reasonable. This could be much smaller, even.
// don't leave the search hanging around
$dispatchProps['timeout'] = 300;

It's a pretty simple search, the auto key/value extraction already gets the q= stuff out of the referer field.
// using head to get only what I want makes the search way faster
$job_id = $searchMgr->syncSearch('search sourcetype="spinnyspinny_access_log" google search | head 3&#8242;, $dispatchProps);

Here's what it looks like in my sidebar:

If you want to see it in action, I have it installed in my personal blog at http://www.feorlen.org. It is pulling statistics about my other site at http://www.spinnyspinny.com, which gets a lot of search engine hits from Google. If you want to test it, search for "spinnyspinny" and some other relevant keywords like "yarn" and you will find my site. Don't go abusing it now, because you know that Splunk will be telling me your IP!

audit.log

New in 3.2, the audit.log records who did what based on what capability was requested from the authorization system. It shows both user-initiated actions like login and automated actions like running saved searches.

Login
07-14-2008 10:59:09.434 INFO AuditLogger - Audit:[timestamp=Mon Jul 14 10:59:09 2008, user=admin, action=login attempt, info=succeeded][n/a]

Running a script
07-14-2008 10:59:12.542 INFO AuditLogger - Audit:[timestamp=Mon Jul 14 10:59:12 2008, user=admin, action=run_script_sendemail, info=granted ][n/a]

Dispatch search
07-14-2008 14:43:39.619 INFO AuditLogger - Audit:[timestamp=Mon Jul 14 14:43:39 2008, user=admin, action=search, info=granted dispatch maxtime=0 maxresults=100 [search sudo | eval sizeof=length(host) ] | outputcsv][n/a]

REST request
07-15-2008 08:21:33.576 INFO AuditLogger - Audit:[timestamp=Tue Jul 15 08:21:33 2008, user=admin, action=search, info=granted REST: /search/jobs][n/a]

license_audit.log

These are the LicenseManager event that used to be reported in splunkd.log, now they are in their own file. The things to pay attention to are quotaExceededCount (number of license violations,) peak (all-time high daily volume) and todaysBytesIndexed. rolloverCount is the number of rollovers since last cleanUsually there is one event generated a day, just after midnight, but there can be others if the instance has been restarted.

07-15-2008 00:01:38.456 INFO LicenseManager-Audit - Audit:[timestamp=1216105298 quotaExceededCount=0, lastExceedDate=0, peak=14699861, rolloverCount=1, totalCumulativeBytesAtRollover=14699861, todaysBytesIndexed=14699861][Jls7bqb2G3dcwAgzAmi0P5pmJn1+IgDwMpoxmW1idMGbA1IlW2amr8tYq5ROlL3bysBxpCV46OEBCt3MJxjI73VvmGSWffU5C+1K3UXYejOLBdinoRavtk+hgLil69eF4n/vQ2mVixK179iHVkzckUcUe8X8iz8qPZT6BEvFhh0AukKlk6IFCrXWRftYysMEIR0IAmcuns7PWBzo/FmEOdm9rBKfVnNMKSvvos39QVooj4O6Km2+xsMUododll8w9IMrl9l0dDHW4AhfZfEN7Sf8krE1c/T/Q+VAxMRgzB0iqJWIddtIxgp6pmdBzD2q7dk9L2pAbkjzDlXRM5GyAg==]

metrics.log

I've talked about this one before, when trying to identify high volume data inputs. New for 3.3, in addition to the default 10 items per period you can configure how many items are reported in metrics.log by setting maxseries in limits.conf. (See limits.conf.spec for details.) Making this number larger will impact performance, but you can do it for investigating a specific issue. Or you can reduce it also. As before, it's a sample of the top n items for each group in a 30 second period. So if you have 200 sources, you won't see all your data inputs here. We are already talking about what metrics we can report, so in 4.0 expect to see new options.

Track blocked queues by looking for "blocked!!":
06-24-2008 09:22:08.792 INFO Metrics - group=queue, name=parsingqueue, blocked!!=true, max_size=1000, filled_count=21, empty_count=0, current_size=1000, largest_size=1000, smallest_size=908

See which processors are actively running:
07-09-2008 14:03:43.876 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0.321082, executes=90770, cumulative_hits=218992256

Diagnostic searches with CLI dispatch

The new dispatch search allows searching across many more events than the older search command. From the CLI, you can use the dispatch command or write something that uses the REST API. Particular searches can tell you more than just returning events. Dispatch from the CLI is particularly suited to this as it's designed for reporting across huge sets of events (although not to return those hundreds of thousands of events.) It may take a while to run, but it will complete.

How many events?
./splunk dispatch "sourcetype=access_combined | stats count"
./splunk dispatch "sourcetype=access_combined starttime::04/25/08:00:00:00 | stats count"

How big are they?
./splunk dispatch "host=foohost1 | eval sizeof=length(_raw) | stats sum(sizeof)"

How big are various other things?
./splunk dispatch "sourcetype=syslog | eval sizeof=length(host) | stats avg(sizeof)"

Note that all of these use additional search commands to report on the set of events rather than the events themselves. Actual results returned from dispatch via the CLI are maintained in memory, so trying to get back thousands of events or more can cause serious problems. Don't do it.

.*?&lt;h2 class="fn"&gt;\s*(.*?):&lt;br /&gt;([^\n]*)\n.*?&lt;div class="info"&gt;\n\[([^&lt;;]*);?\s*(\d*)\]?.*?&lt;span class="rating"&gt;(.*?)&lt;.*?&lt;div class="content description"&gt;(.*?)&lt;/div&gt;.*? - &lt;span class="reviewer"&gt;&lt;span class="vcard"&gt;&lt;span class="fn"&gt;(.*?)&lt;/span&gt;.*?title="\d+"&gt;(.*?)&lt;

I can now run some interesting queries :

• * | chart avg(rating) by releaseYear
  
  Which graphs the average rating per calendar year of the release.
• *| stats count(title), avg(rating) by artist | search "count(title)">2| sort "avg(rating)" d | head 10
  
  This shows the top rated artists that have a least 3 reviews on pitchfork
• * rating<=10 rating>0 | stats avg(rating) as avg_rating, count(title) as title_count by label | search title_count>3 | sort avg_rating | head 10
  
  This shows that Invisible Records are the worst reviewed label on Pitchfork.
• * | stats count(title), avg(rating) by reviewer | search "count(title)">4 "avg(rating)">7.5 | sort "avg(rating)" d
  
  This search finds all the reviewers that have at least 5 reviews and on average score higher than 7.5. So if you want a good review on pitchfork you're better off with Luke Buckman
• * | eventstats count(title) as titleCount by reviewer | search eventtype=7_dirty_words titleCount>3 | stats count(title) as ct ,max(titleCount) as mf by reviewer | eval blue_index=ct*1.0/mf | sort blue_index d
  
  This is my personal favourite, it's a list of reviewers most likely to use the one of George Carlin's seven dirty words (nsfw). The mf column is the count of reviews with one of the words and the ct row is the review count for that reviewer. The blue_index is the mf/ct.

So there you go : Splunk > it's not just for logs.

Old CLI search:

./splunk search "sourcetype=access_combined googlebot | stats count" -maxresults 500
count
- -
213

New CLI search:

./splunk dispatch "sourcetype=access_combined googlebot | stats count"
count
- -
213

While the results look the same for this simple search, there is a lot different going on behind the scenes. The search command needs to load all the events it touches into memory, so there is only so much of the index it can search at one time. The data generation part, before the pipe, will only return maxresults number of events, which may not be all of them. If you then filter with additional search commands you won't get all of what you think you should. You can increase maxresults (default for the CLI is 100) but you can only push it so much until you run into memory problems.

The dispatch search kicks off a job that runs until completion, no matter how long it takes. But one thing to keep in mind is that CLI dispatch is designed for reporting: the actual results are all in memory so you can't get back thousands of results from a single search. Use reporting commands like stats or narrow your searches so they won't have more than a couple hundred results. (If you need more, write something that uses the REST API where you have access to job control.)

So how this applies to alerting:

In the UI, when a scheduled search runs, it uses a search command to actually generate the alert. There are a couple different ones, but as most people want an email I'll focus on sendemail. (Docs here: http://www.splunk.com/doc/3.3/user/UnsupportedCommands#sendemail.)

Any search can use the sendemail search command, it's not limited to the UI. So I can do this:

./splunk dispatch "error | sendemail to=sysadmins@example.com from=splunk@example.com"

This runs the search and then looks for a mail server (by default on the local machine) to send the message. Since it's using dispatch, you can kick off a bunch of these and they will all run independently of each other. You can look at the jobs from the REST endpoint:

https://localhost:8089/services/search/jobs

Splunk Atom Feed: jobs
Updated: 2008-07-14T10:39:16-0700 Splunk build: 38343
dispatch
cursorTime 1969-12-31T16:00:00.000-08:00
error
eventCount 316
isDone 1
isFinalized 0
isPaused 0
isStreaming 0
keywords sudo
resultCount 100
sid 1216057125.31
ttl 3570.9 seconds
events - results - timeline - summary -
control:

2008-07-14T10:38:47.000-07:00 | admin

Here's an example I set up on my local machine, an OS X 10.5 box which uses postfix. I've already made sure postfix is running and I can receive mail to my local account.

I wrote a script that does 50 searches, all set to alert with an email address. Note the auth in the command, if you aren't already authenticated you will need to use the auth command as part of the CLI search. In a production environment, you would want a more sophisticated means of handling login credentials than sticking plaintext into a script. (You could also use a restricted user created only for CLI searches.)

[root]:/opt/splunk3.3/bin$ more alert_overload.sh
./splunk dispatch "sudo | sendemail to=feorlen from=foo01&#8243; -auth admin:changeme&#038;
./splunk dispatch "sudo | sendemail to=feorlen from=foo02&#8243; -auth admin:changeme&#038;
./splunk dispatch "sudo | sendemail to=feorlen from=foo03&#8243; -auth admin:changeme&#038;
./splunk dispatch "sudo | sendemail to=feorlen from=foo04&#8243; -auth admin:changeme&#038;
./splunk dispatch "sudo | sendemail to=feorlen from=foo05&#8243; -auth admin:changeme&#038;
./splunk dispatch "sudo | sendemail to=feorlen from=foo06&#8243; -auth admin:changeme&#038;
./splunk dispatch "sudo | sendemail to=feorlen from=foo07&#8243; -auth admin:changeme&#038;
[...]

When I run this script, it starts up all these searches. (Note that each one starts up another python! Keep that in mind.) When they complete, they send an email alert.

N 16 foo13@AndreasSplunkP Mon Jul 14 10:59 393/280230 "Splunk Results"
N 17 foo17@AndreasSplunkP Mon Jul 14 10:59 393/280230 "Splunk Results"
N 18 foo11@AndreasSplunkP Mon Jul 14 10:59 393/280230 "Splunk Results"
N 19 foo32@AndreasSplunkP Mon Jul 14 10:59 393/280230 "Splunk Results"
N 20 foo09@AndreasSplunkP Mon Jul 14 10:59 393/280230 "Splunk Results"
? s* dispatch_test.mbox
"dispatch_test.mbox" [New file]
? x
AndreasSplunkPowerbook-2[feorlen]:~$ grep ^From: dispatch_test.mbox | wc -l
50

The messages don't arrive in the same order, but they do arrive. For these 50 test searches, it was about 20 seconds for all of them. More complicated searches will take longer. One thing to know is that if you are searching faster than it can complete, as in every minute you start a search that takes two minutes to run, they will back up and take a while to complete. There is no hard guideline, as it depends on the individual searches and the overall load on the instance.

In the latest releases, we have search-time discovery of transactions, with the new transaction search command. Transaction collapses a set of events that belong to a transaction into a single event. You can specify the parameters as arguments to the transam operator right in the search, or you can refer to a named-transaction definition in transactiontypes.conf. A few simple examples will give you an idea of some things you can do.

• get events with 'http', and group any search results into "bursts" of events, grouping any events that occur within two seconds of each other into the same transaction event. [Note: there is an implied "search" command at the head of all searches, so "http" is really "search http".]

http | transaction maxpause=2s

• get events with 'http', and collapse those that share the same host and cookie value, that occur within 30 seconds:

http | transaction fields=host,cookie maxspan=30s maxpause=30s

• get events with 'sendmail', and collapse those that have the same userid, between a login and a logout, that occur within 10 minutes:

sendmail | transaction fields=uid startswith="eventtype=login" endswith="eventtype=logout" maxspan=10m maxpause=10m

• get events with 'http', and then find transactions as defined by email_transaction found in transactions.conf:

http | transaaction email_transaction

• Find transactions that change a password, near where there were unsuccessful root logins. To break it down - search for unsuccessful root logins, find time ranges around those root logins, find transactions in those those regions, and finally look for password changes in the transaction.

  root login NOT fail*
  | localize maxspan=1m maxpause=1m
  | map search="search starttimeu=$starttime$ endtimeu=$endtimeu$
  | transaction session |  search password change"

• rpm -ivh http://fpdownload.macromedia.com/get/flashplayer/current/flash-plugin-9.0.124.0-release.i386.rpm
• yum install nspluginwrapper.{i386,x86_64} pulseaudio-lib.i386
• yum install flash-plugin
• yum erase rhythmbox.*
• mozilla-plugin-config -i -g -v
• mozilla-plugin-config nspluginwrapper -i /usr/lib/mozilla/plugins/libflashplayer.so

(Optionally, if you haven't imported the Adobe GPG key, you will have to run the following command)

• #rpm - import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux

Forwarding metrics.log

Forwarding metrics.log will require that you make the following changes to the configuration on each Splunk instance that you would like to collect the metrics from:

If you have many Splunks in your environment, then making these changes on each one of them manually is certainly not an option you would cherish. This is where Deployment Server can help you centralize all your configurations in one place and distribute them to all or selected instances.

Here's something I like to do

1. Have all Splunks point to a common Deployment Server

This can be achieved very easily by creating/editing deployment.conf in $SPLUNK_HOME/etc/system/local on each Splunk instance.

[deployment-client]
deploymentServerUri=&lt;your_deployment_server_uri&gt;:&lt;mgmt_port&gt;

For some of my distributed testing on EC2, I have images that include this configuration in the default image (AMI). Using this approach guarantees that configurations never ever have to be changed by hand!

2. Create a bundle

Create a bundle by any name (I called it deployable) and make sure it is available in your Deployment Server's serverClassPath. This bundle should have two files - inputs.conf and outputs.conf - as described above - here's a sample bundle you could re-use.

3. Make the bundle available to all Splunks

Make all deployment clients that connect to the deployment server to be part of the deployable service class. This is achieved by changing deployment.conf on Deployment Server again as:

[distributedDeployment-classMaps]
*=deployable

4. Refresh Deployment Server Configuration

This CLI on your Deployment Server instance will make it aware of the new configuration without a restart:

splunk reload deploy-server -auth admin:changeme

You are now all set and all Splunks in your environment will automagically download and apply the bundles within a minute! And in another 30 seconds, your Deployment Server will start aggregating metrics information about your entire data-center!

We want to hear about your experiences in managing Splunk - use the Comments below or send me an email directly at inder@splunk.com.

Forwarding-side

Splunk uses a component called TcpOutputProcessor, which is configured using outputs.conf, to forward data to another Splunk or non-Splunk entity. This is something that a lot of people also refers to as a forwarder. Each TcpOutputProcessor instance publishes metrics events every 30 seconds - all the fields of these events are described below:

• group=tcpout_connections - this field discriminates this event as being a TcpOutput metric.
• tcpout_group_name:destIp:destPort - the load-balanced group that this metric belongs to. If you have multiple groups defined, a separate event is published for each of those groups.
• host metadata - is always available in an event, and refers to the host on which the forwarder is running.
• sourcePort - the local port that is used to connect to the remote entity.
• destIp - the ip address of the remote server to which events are being forwarded.
• destPort - the destination port on which events are being forwarded.
• tcp_bps - bytes per second averaged over last 30 seconds.
• tcp_kbprocessed - total KBytes processed since this connection went live.
• tcp_eps - events per second averaged over last 30 seconds.
• tcp_dropped_events - number of events dropped on this connection.

Indexing side

Similarly on the indexing side, if you have configured inputs.conf to receive data from one or more forwarders, a metrics event is published every 30 seconds for each connection into your indexer. All the fields of a metrics event on the input side are described below:

• group=tcpin_connections - this field discriminates this event as being an input metric.
• sourceHost - The hostname of the entity that is forwarding data to this indexer. If hostname is not available, then it's IP address is used.
• sourcePort - The remote port of the forwarding entity.
• destPort - The local port on the input side for which this metric is being collected. Typically this port is defined in inputs.conf.
• tcp_bps - bytes per second averages over last 30 seconds.
• tcp_kprocessed - KBytes processed since the connection was established.
• tcp_eps - Events per second averaged over 30 seconds.

These metrics will now enable you to get unusual insight into the operation of your forwarders and indexers. Here's a sample query that you can run on each indexer instance to get a report on thruput by each forwarding entity:

index=_internal metrics "group=tcpin_connections" | timechart span=30s avg(tcp_bps) by sourceHost

Also, I created a saved search, and used Splunk's reporting features to always show me the current status on a dashboard.

Now that you have all of this nice data, I am sure you would like it all aggregated in one location.

Good luck playing with these metrics, and if you have any suggestions on what more you would like to see, drop me a line at inder@splunk.com.

Wanted to take a minute to talk about authenticating Splunk against Active Directory. In case you didn't know Active Directory is running on top of LDAP. While the guys up in Redmond do their best to make sure tha you have no need to know LDAP they give you the ability to interface with it over LDAP if you know what you're doing. Let's take this time to let you know what you need to do.

If you are comfortable with the command line you can run the command ldifede. The ldifde command is the windows equivalent of ldapsearch and should allow you to get an ldif entry for yourself and a group. With those two entries we should be able to come up with authentication.conf that will allow Splunk to authenticate users.

For those of you that are more comfortable with a GUI The Sysinternals team offers a nice utility called Active Directory Explorer. This gives you tree view of your Active Directory/LDAP structure.

The information provided from these utilities is pretty much everything you need to know in order to follow along with the documentation. If you are still struggling to get it working send an email to support@splunk.com with the output from the ldifde command and your authentication.conf and someone from team will help square you away.

As one of the Splunk Support Monkeys I am going to try to start a semi-regular series of posts on a topic that is near and dear to me - getting the Splunk community to be able to troubleshoot their issues without the need to reach out to the Support Team.

The most important piece of any troubleshooting exercise is getting a solid understanding of the problem. The common statement "Shit is broke" while 'summarizing' the problem doesn't do much in the way of isolating the specific problem. Taking a minute or two to think about the problem at and documenting the sequence of events leading up to the problem goes a long way to getting outsiders up to speed on the issue.
Here are few things to keep in mind when working with support:

I don't work in the next cube over.

This means I don't have insight into all of the other moving parts of your network. Try avoiding acronyms that are specific to your organization. I don't know the naming convention that you use for machine names, so if one box is in LA and the other is New York tell me, don't expect me to know that foo.company.com is sitting in the LA data center.

Less is not more.

You can never give a support engineer to much data. Often times folks think that they have identified the offending error message in the logs and provide that one line in their support ticket. The problem with this is that the support engineer does not get the benefit of context. Most errors are the result of a series of events leading up the final failure. Being able to see what was going on leading up to the problem often times is what allows us to identify cause. The basic rule of thumb is if you think it would be at all useful share. If I can channel Don Rumsfeld for moment: It easy to know what you know, it is hard to know what you don't know.

Reduce the problem to the fewest number of variables possible.

Remember your 7th grade Algebra class and those complex equations that Mr Buckner had you had solve? You started off solving for x and then you went back using your knowledge of x to determine the value of y. The same is true when troubleshooting software. When you try to solve 4 problems at once you end up polluting your results; you can't tell if the change you made for x resulted in y blowing up. By breaking the problem into smaller chunks you are operating in a more scientific manner and the results have more credibility.

Log like there is no tomorrow.

Debug logs are your friend. In normal operations the logs don't need to be verbose but when you are trying to figure something out why not give yourself the benefit of the secret messages that the developer put in the code for precisely this reason. It is also helpful to push the existing log file out of the way when starting in a debug mode. While I said early that you can never give a support engineer to much information the majority of the stuff in your logs (especially if you've been running for awhile) is going to be white nows. Starting in debug mode with a fresh log means that the problem and the only the problem are going to be in the log.

From the standpoint of configuration and what WMI is capable of doing, in the context of Splunk, WMI can be used in two ways: to pull event logs and to query instrumentation data. Assuming that you have enough credentials to poll event logs agentlessly, you can simply specify host name and the log file you are interested in. This is an example of retrieving "Application" event logs from a remote machine named "remotehost":

[WMI:RemoteApplication]
namespace = \\remotehost\root\cimv2
interval = 10
event_log_file = Application
disabled = 0

The other aspect of WMI warrants more explanation. To get data from WMI providers, you query them using WQL (WMI query language), which is a subset of SQL. Simply specify a query, and all fields returned by the provider will be automatically collated as an event. (Some queries return multiple results, and hence generate multiple events.) An example query will be select FreeMegabytes from Win32_PerfFormattedData_PerfDisk_LogicalDisk, which will poll free disk space from all logical disk partitions on the system.

This is an example config setup that gets runtime information for all running processes on a local machine every 30 seconds:

[WMI:LocalAllProcesses]
namespace = \\.\root\cimv2
interval = 30
wql = select * from Win32_PerfFormattedData_PerfProc_Process
disabled = 0

With this you can easily chart memory usage by process.

The default install of the preview includes several preset performance queries. If you look at %SPLUNK_HOME%\etc\system\default\wmi.conf, you will find three default config stanzas. To see a list of what all is available for querying, google for "WMI classes" and browse the MSDN documentation. There is tons of stuff that you can splunk, including detailed memory usage, network utilization, disk usage, detailed process runtime information. Also, take a look at the WMI documentation.

Happy Splunking with WMI.

http://www.splunk.com/index.php/preview

I want to introduce to you one the latest features for Windows Splunk - the monitoring of Windows registry in real time for activity/events, and the indexing and searching these events with Splunk.

While working on this we had a few challenges:

First, there aren't any published win32 APIs that does this in user mode. The best that you can do with win32 API is to poll the registry for certain registry key/hives, and you'll be notified when if the key or subkey of the hive has been changed. Even when you get a notification for a change, you will not be told which key exactly has changed, you'll have to figure that out yourself .

Second, scalability. You can't possibly poll all of the registry in user mode for changes. There are simply too many keys to query.

The solution is to write a device driver that hooks to the kernel and intercepts all registry events. The driver bubbles up the events to the user mode for filtering and tagging, and finally pipe them to Splunk for indexing. Obviously, this driver needs to be very stable and reliable, needs to scale to the point where if you want to monitor all of the events in the registry, and it should be able to handle the load.

With this preview release we launched the first version of the splunk-regmon tool. The tool writes events to standard output, and using Splunk's ExecProcessor(popen). Splunk is able to get these events and send them through the indexing pipeline. A basic filtering is in place, hard coded for now to only monitor registry events related to changes - i.e. Create, Delete, Set, etc. Create type events are represented by "CreateKey" reg_event field, Delete by "DeleteKe" and all of the Set event eg: SetValueKey, are represented by SetKey reg_event field. In our next release this filtering will be configurable.

Here is what a windows registry event looks like with Splunk:

Drop us a note and let us know what you think of this new feature and any concerns you may have, or ideas of how we can make it better.
How would you use it and how it would be useful to you?

As one of the splunkers responsible for answering the phone I'm going to use this space to talk about something near and dear to my hart - empowering my customers so they are able to figure out their own problems thereby allowing me read FARK all day long.

Since we recently released our Windows version a bunch of the folks in the office have been trying to figure out how they do the things they do in a UNIX enviornment (like wget a file) in Windows. I've been sharing some of my favorite Windows resources here at the office and figures the rest of you would probably like to know about them as well.

Google
Everyone seems to start here when they are looking for something. Most however don't know that http://www.google.com/microsoft will restirct your search to Windows sites. They also have these search sites for linux, bsd, and the mac.

SysInternals
Mark and Bryce have created the ultimate coolection of free Windows utilities. Simple executables that allow to get so many of the diagnostic/monitoring things that a UNIX admin takes for granted. Some of my favorites (and especially useful in working with Splunk) in no particular order:

• AccessEnum
  Lets you see who has access to what. This is really helpful when trying to figure out why Splunk isn't indexing one of your files.
• Process Monitor
  Watch the registry, running process/thread/DLL, and file system usage in real-time
• PS Tools
  A bunch of command-line utilities for listing the processes running, working with the event log, rebooting the machine, etc.
• Active Directory Explorer
  Advanced viewer/editor for Actiive Directory. This will be a godsend you are trying to configure Splunk to authenticate against your domain controller
• WhoIS
  Doesn't do much in the way of troubleshooting Splunk, but who doesn't want to be able to see if ultramegaextrmeme.com is available and if not who the lucky owner is? BTW it is available.
• TCPView for Windows
  Lets you see all the TCP and UDP endpoints on your system, including the local and remote addresses and state of TCP connections.

Hope that helps you guys out. All of you experienced Windows folks if you've got others out that there post to the comments. If my jaw hits the desk when I click the link I will send you a Splunk koozie.

Thanks
e.

The easiest way to get started with the client library is to get into Splunk's Python environment. Locate your Splunk install directory (/opt/splunk by default), and start the python interactive shell that comes with Splunk:

# bin/splunk cmd python


This will launch the interactive Python prompt, which starts off looking like this:

Python 2.5.1 (r251:54863, Nov 18 2007, 16:13:41)
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>


Starting a search

Import the Splunk modules:

import splunk.auth
import splunk.search as se


If you have installed Splunk with the default settings, then your hostpath is https://localhost:8089. The client library knows this default, so you can authenticate directly by providing a username and password:

key = splunk.auth.getSessionKey('admin','changeme')


The getSessionKey method automatically caches the session key in the current interactive session, so you don't have to pass it along to subsequent methods. In a production implementation, or if you are connecting to multiple servers, you'll need to keep track of separate session keys.

If your server is on a different hostname or port, then you need to first update the session defaults:

splunk.mergeHostPath('splunk_hostname:12000', True)
key = splunk.auth.getSessionKey('admin','changeme')


The mergeHostPath method takes host information in many different forms:

• hostname
• hostname:port
• https://hostname
• http://hostname:port

Next, start a search:

job = se.dispatch('search error')


This creates a search job handle object job and start a running search on the server for events that contain the term "error". If you are connecting to multiple servers, then you'll also need to provide hostPath and sessionKey parameters as well. This handle is keyed off of the search job ID that is generated by the server, and is available via:

job.id


With this ID, you can always use your web browser to check on the status of a particular job by opening up:

https://localhost:8089/services/search/jobs/12345

where 12345 is the ID that you just generated.

There are a few properties on the SearchJob object that will be of immediate use:

• job.isDone - a boolean value that indicates if the search has completed
• job.count - the number of events that have been matched against the search
• job.cursorTime - the current position of the search cursor; when dispatching a search, the cursor moves in a reverse chronological order

Working with search results

The raw events are the original event data that were indexed by Splunk, according to the data input rules. They are available as an interable container object:

job.events


This object works just like a list, and you can iterate and slice it to obtain events. The events are stored in reverse chronological order.

for x in job.events:
print x


This code will iterate over every event returned in the search and print out the raw text, which could be every event in your index if you so choose. The iterator will begin returning data as soon as it receives the first event, and will continue until the isDone property is True.

You can also retrieve specific rows of data using the standard python slice operator:

job.events[2] # returns the 3rd event in the search results
job.events[2:10] # returns events 3 through 10 as a list
job.events[-1] # returns the last event in the results


The items returned by iterating or slicing are actually Result objects that have additional properties:

• job.events[0].raw - the raw event text (the same value as print job.events[0])
• job.events[0].time - the event timestamp, as a datetime.datetime object
• job.events[0].fields - a dict of all the fields associated with the event

For example if you wanted to see the host field for an event:

job.events[0].fields['host']


Or if you wanted to see all of the host entries for each event:

for x in job.events:
print x.fields['host']


Or alternatively, in shorthand:

for x in job.events:
print x['host']


If you want to print out a human-readable timestamp for events that came from the 'firewall' sourcetype:

for x in job.events:
if x['sourcetype'] == 'firewall':
print x.time.ctime()


When you are finished with the search job, remove it from the server by calling:

job.cancel()


Otherwise, the job will persist on disk until the specified timeout (TTL), which is 24 hours by default.

Enter the Atom standard: a standards-track schema that defines a generic collection/item container format in XML. Most people equate Atom to an RSS competitor, which is true, but that only covers half of what it does. The Atom Publishing Protocol is a well-defined protocol for performing CRUD (Create, Read, Update, Delete) operations on items over HTTP. The Atom Syndication Format, which is the most commonly used portion, defines the XML schema used to deliver data during a Read operation. Atom was spearheaded by Sam Ruby, and is now back by people like Brad Fitzpatrick, Tim Bray, Jeremy Zawodny, Mark Pilgrim, and is heavily implemented by Google.

Like most software systems, the majority of Splunk's internal entities can be loosely viewed as a collection of similar items. The requested searches, configuration information, saved searches, users, roles - all just collections. So instead of creating five separate XML schemas for each of these collections that perfectly describe their contents, I chose Atom to serve as a single generic container to describe all of the entities. This kind of reuse is echoed by Pat Helland of Amazon, who gives a great talk on relating the rise of the industrial age to standardization, and Tim Bray (Mr. XML himself), who advocates against creating your own XML unless absolutely necessary.

The benefit of sticking to a standard is that there is a much greater chance that external developers already know exactly how to consume your data with very little work. Not only are language-level Atom parsers available everywhere, but entire applications have been specifically built to consume Atom. For instance, here's a screenshot of the NewsFire feed reader displaying all of the searches that exist on my local Splunk server:

All I had to do was to supply a URI and login to NewsFire, and then it took care of the rest. No XSLT, XPath, or custom DOM iteration necessary; it just works. As far as I know, Splunk is one of a handful of enterprise companies that has integrated Atom at such a core level. Hopefully, for you it means that there is one less bucket of tag soup you have to deal with, and one better product that you enjoy using.

Each event is represented by a single square particle that flows from its place in a legend of values to its corresponding position in a stacked column chart. Upon landing in the column chart, one of the event's fields is output in a readable format below the chart. Both the legend of values and the stacked column chart retain the order of their values according to a configurable comparator and truncate older values to make space for new ones. Rolling your mouse over any column displays the field values for that column.

Replay currently consumes csv files and is configurable through an xml file. The current demo charts twikipage edits split by twikiuser (both sorted alphabetically) and outputs truncated raw events below the chart. The simulated event stream is running at a rate 2000 times real time.

I'm currently working on getting Replay hooked directly to Splunk's API and building out interface elements so that it can be configured visually.

You can check out the wiki page on Replay over at Splunk's developers wiki.

With 3.2, we have begun moving some of Splunk's core services over to a proper REST API. Now, for those of you who have already been using the REST API in 3.1, the new API in 3.2 and beyond is distinctly different, and is intended to replace any older versions. Therefore, the REST API of version 3.1 and before will now be referred to as the UI API, and the term "REST API" will refer to the new API that I'm covering in this post.

Before I dive into the details though, I'd like to clarify the usage of "REST" and what I mean when I speak of it. First of all, REST is not a protocol or standard. There is no RFC, or ISO specification on what constitutes REST; it is a philosophy about the relationship between entities in a software system and the interface to interact with those entities. Roy Fielding's original thesis named it Representational State Transfer, which when put into practice means that URIs should convey meaning in a durable manner. In essence, REST emphasizes the "what" of a system rather than the "how". In comparison, SOAP interfaces are based on codified standards that dictate the communication protocol. Lots more information on REST can be found on Wikipedia or in book form as RESTful Web Services by Leonard Richardson.

The New Search API

Splunk's new search interface allows for multiple searches to be scheduled concurrently, and for the results to be retrieved asynchronously. Assuming that you've installed Splunk using the default settings, you can see all of your search jobs by pointing your browser to:

https://localhost:8089/services/search/jobs

This returns an Atom feed of all the search jobs present in the server. Each job has an ID, and so the URI for the Atom entry of a search job of ID=1234 can be found at:

https://localhost:8089/services/search/jobs/1234

Following that RESTian schema, each facet of a search job can be found as a sub-endpoint as well. For instance, the events, results, timeline, and summary data for each search can be found at:

https://localhost:8089/services/search/jobs/1234/events
https://localhost:8089/services/search/jobs/1234/results
https://localhost:8089/services/search/jobs/1234/timeline
https://localhost:8089/services/search/jobs/1234/summary


Each of those endpoints returns data in XML format by default, but can be switched over to JSON or raw text format.

The key to implementing a successful REST API lies in using the HTTP protocol to its fullest potential. Instead of adding a new search via something like /search/add_search, we simply POST to the parent /services/search/jobs endpoint. Instead of adding an extra /search/delete_search endpoint to delete a job, you issue an HTTP DELETE command directly on the /services/search/jobs/1234 endpoint. By treating each endpoint as a direct entity mapping, we simplify comprehension and dramatically reduce the total number of discrete endpoints.

The configuration API

The same model applies to our configuration system as well. Splunk stores its configuration in conf-style text files, using traditional stanza-separate key/value pairs. For example, the server.conf looks like:

[httpServer]
atomFeedStylesheet = /static/atom.xsl
max-age = 3600
follow-symlinks = false


To access this file from the API, you would first browse to:

https://localhost:8089/services/properties/server

This endpoint returns an Atom feed of all of the stanzas contained in the file. To view all of the key/value pairs in the [httpServer] stanza, browse to:

https://localhost:8089/services/properties/server/httpServer

To read a single key value like max-age, browse to:

https://localhost:8089/services/properties/server/httpServer/max-age

To change that value, issue an HTTP PUT to the same endpoint. To add a new key, issue a POST to the stanza-level endpoint, or issue a PUT directly onto the new key name.

The advantages of exposing everything via HTTP are obvious when it comes to integration and remote management. Every modern programming environment speaks HTTP, which means you can programmatically interact with Splunk from wherever you want. Everyone also uses a web browser, which means that probing the API is as easy as browsing the web.

Even with a simple API, there's no reason for developers to recreate a language-specific library to access Splunk so we're working on releasing a few downloadable libraries for use in Python, .NET, Java, and Perl. Check the Splunk Labs page for more information about those.

Once configured and connected, you start the server and accept connections.

I then grabbed the IO-LCDproc perl module and modified it to display to the LCDproc server. You can get the IO-LCDproc through CPAN.

Here is the code that would go in your $SPLUNK_HOME/bin/scripts directory

[source:python]
#!/use/bin/perl -w
use IO::LCDproc;
use IO::Socket;
use strict;

&amp;usage if (! $ARGV[0]);

my $client = IO::LCDproc::Client-&gt;new(host =&gt; "localhost", name =&gt; "MYNAME", port =&gt; "13666&#8243;);

my $screen = IO::LCDproc::Screen-&gt;new(name =&gt; "screen");

my $title = IO::LCDproc::Widget-&gt;new( name =&gt; "date", type =&gt; "title");

my $first = IO::LCDproc::Widget-&gt;new(
name =&gt; "first", align =&gt; "center", type =&gt; "string", xPos =&gt; 1, yPos =&gt; 2,
data =&gt; "test");
my $second = IO::LCDproc::Widget-&gt;new(
name =&gt; "second", align =&gt; "center", type =&gt; "string", xPos =&gt; 1, yPos =&gt; 3
);
my $third = IO::LCDproc::Widget-&gt;new(
name =&gt; "third", align =&gt; "center", type =&gt; "string", xPos =&gt; 1, yPos =&gt; 4
);

$client-&gt;add ( $screen );
$screen-&gt;add ( $title, $first, $second, $third );
$client-&gt;connect() or die "Cannot Connect: $!";
$client-&gt;initialize();

$title-&gt;set( data =&gt; "Splunk2LCD" );
$first-&gt;set( data =&gt; "$ARGV[1]" );
$second-&gt;set( data =&gt; "$ARGV[4]" );
$third-&gt;set( data =&gt; "$ARGV[5]" );

sleep 5;
exit 1;

sub usage {
print &lt;&lt;USAGE;
LCDproc Client for Splunk
Mark Cohen
Usage: ./splunk2lcd2.pl ARGV0 ARGV1 ARGV2
USAGE
exit 1;
}
[/source]

Browsing over on Wikipedia, one excerpt states that "a platform describes some sort of hardware architecture or software framework", and the description for a software framework, says it "may include support programs, code libraries, a scripting language, or other software to help develop and glue together the different components of a software project".

A platform can be considered as a type of framework - one which helps developers write software faster by a) giving them the tools to develop against it, and b) transparently dealing with the under-the-hood, nitty-gritty work necessary when dealing with difficult problems. Difficult problems like indexing and searching gigabytes upon gigabytes of event data, for example.

Well, that's exactly what the Splunk Platform does for developers. It provides resources, examples, and SDKs for developing a variety of applications around the robust Splunk engine, and it provides a launching point for domain specific development, from availability and security, to business intelligence and compliance.

BTW, this isn't something we're just waving our hands around about and saying "look at this white paper, isn't this a nice idea?". Nope. Platform is here, and it's here today, with links to real code, real content, and real resources for the developers looking to write the next great idea.

Here's how to get started writing your first application against the Splunk Platform:

1. 1. Download a Preview of Splunk that has a brand new shiny REST-based API built into it.
2. 2. Head on over to the developer's wiki and start digging around in the API howtos.
3. 4. Download the new .NET SDK from its Google Code project page.
4. 4. Join the any of the projects and start contributing code/content.
5. 5. Join the new Splunk Labs list and start interacting (asynchronously) with our developers.
6. 6. Hop on #splunk on IRC and chat with us in real time.

We'll be continuing to add content and resources to the Platform effort, and we encourage your participation in the development community as it forms.

Observations
1. Header-body. Some applications, for different reasons, choose to format their log files using a header and a body section. The header usually describes the way the fields are organized in each logged event, while the body consists of logged events, usually one per line, with field values delimited as described in the header. W3C, CSV etc come to mind, see examples
2. Single-delimiter. Other applications choose to use a single delimiter to delimit keys from values and values from keys, while this is not very common it's been observed in the field.

Data Examples
The following header-body sample, as you can probably guess, is from an exchange server. There is a header section which among other things has the list of field names, delimited from each other using the delimiter used to delimit values in the body section, in this case a tab character is used (even though our blogging platform chooses to mangle tabs to spaces - gotta love it !!!).

# Message Tracking Log File
# Exchange System Attendant Version 6.5.7638.1
# Fields: time client-ip cs-method sc-status
14:13:11 10.1.1.9 HELO 250
14:13:13 10.1.1.9 MAIL 250
14:13:19 10.1.1.9 RCPT 250
14:13:29 10.1.1.9 DATA 250
14:13:31 10.1.1.9 QUIT 240


The following example shows how a single-delimiter can be used to list fields, it is pretty easy for us, as humans, to recognize the key value pairs:

"url http://splunk.com referer http://dev.splunk.com ip 10.10.10.10"


Enabling header-body kv/extract
The delimiter based KV extraction solves the header-body problem by adding the capability to assign field names to extracted values by doing single-level tokenization/splitting (ie single delimiter) instead of the normal two-layered one described earlier. Unfortunately, however, this is only available through transforms.conf* and it requires manual specification of the field names (no automatic field name detection). To this end, we introduce another transforms.conf configuration variable, defined as follows:

FIELDS = <quoted string comma/space separated list>
- List of names to associate with each extracted field value. The first entry is associated with the first
field value, the second with the second value and so on...

Example from above data:
FIELDS= "time", "client-ip", "cs-method", "sc-status"


Thus to enable header-body KV extraction one needs to specify one delimiter and a list of fields to attach to each extracted value. Let's walk through the MS Exchange sample data: (1) we know the field delimiter is the tab character and (2) the field list, in their correct order, is in the header of the file all we have to do is quote the field names. The configuration stanza in transforms.conf should thus look like this:

....transforms.conf....
[exchange]
DELIMS = "\t"
FIELDS = "time", "client-ip", "cs-method", "sc-status"


To apply this transformation you can then run ".... | extract exchange reload=t auto=f| ....", there's no need to restart the server after editing the transfroms.conf as long as "reload=t" is specified in extract (btw auto=f turns off automatic KV extraction)

The results of this transformation ,on one of the events, would then be:

"14:13:11 10.1.1.9 HELO 250"

time=14:13:11
client_ip=10.1.1.9
cs_method=HELO
cs_status=250


Easy huh!? Try it in your data, we'd love to hear back ......

*The reason why this is only available through the configuration is that amount of configuration information needed.

Enabling single-delimiter kv/extract
There's yet another trick in the delimiter KV extraction - the single-delimiter extraction. Single delimiter extraction pairs extracted field values into key=value as follows: value1=value2, value3=value4 and so on... To enable this extraction via the command line set kvdelim and pairdelim to the same value, for the above example data the extract command should look as follows:

.... | extract kvdelim=" " pairdelim=" " auto=f | ....


To enable single-delimiter extraction via transforms.conf you can either specify one delimiter or two identical delimiters in the DELIMS config variable, thus the following two transforms.conf stanzas are equivalent to each other and to the above command:

....transforms.conf....
[single-delim-1]
DELIMS = " "

[single-delim-2]
DELIMS = " ", " "


The results of these extractions for our sample data would be:

"url http://splunk.com referer http://dev.splunk.com ip 10.10.10.10"

url=http://splunk.com
referer=http://dev.splunk.com
ip=10.10.10.10


NOTE: do not specify a FIELDS variable for the single-delimiter extraction because that will enable header-body extraction.

Thoughts, ?, ideas, comments are always welcomed....

Observation

Most logged events usually contain a list of key-value pairs (e.g. attribute list, method call values etc) in a context-dependent well-defined format. An example of well-defined format: " key-value pairs are separated from each other using ';' while the key is separated from the value using '=' ". More generally, well defined attribute listing formats are not confined to logging, they're part of every event-driven, flexible attribute order, application: e.g. URL get parameter list, HTTP request/response headers, email headers etc... In most application the delimiters are single characters which are least likely to be part of the key or value, whenever the key/value contains any of the delimiters it is normally enclosed in literal-defining characters usually double-quotes (").

Definition: delimiter based KV extraction
Let's first define three character classes:
1. [pairdelim] - non-empty list of characters used to separate key value pairs from each other. (chars after value, before next key)
2. [kvdelim] - non-empty list of characters used to separate the key from the value. (chars after key, before next value)
3. [quoter] - list of characters used to enclose a literal - currently *only* quotes are supported and this variable is not configurable

Thus we can formally define a key-value pair list as follows:

kvlist = <key>[kvdelim]<value>([pairdelim]<key>[kvdelim]<value>)*
key = <string>|<quoter><string><quoter>
value = <string>|<quoter><string><quoter>
quoter = "


Thus, delimiter KV-extraction can be achieved by a two layer tokenization/splitting process:
1. Split on the pair delimiter to extract candidate KV pairs
2. Split on key-value delimiter to separate key from value

Examples:
1. URL - the following is an example of finding what are the delimiters for parameters listed in the query part of the URL. Note, since the query starts after the '?' character - the '?' qualifies as a key-value pair delimiter since it is before the first key.

data = "http://usasearch.gov/search?input-form=firstgov&v%3Aproject=firstgov&query=splunk+it&affiliate=uspto&x=0&y=0"
-------------
pairdelim is "?&" - # parameters in the query are separated by '&', the query starts after '?'
kvdelim is "=" - # variable names are separated from their values using '='


2. HTTP response header:

data = "GET / HTTP/1.1
Host: dev.splunk.com
Connection: close
User-Agent: Web-sniffer/1.0.25 (+http://web-sniffer.net/)
Accept-Encoding: gzip
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Referer: http://web-sniffer.net/"
-------------
pairdelim is "\r\n"
kvdelim is ":"


That was easy, why don't you try it on the following data?
Note: data_hard is all one line however our blogging software sucks at displaying long lines

data_easy = "May 4 14:47:28 gwrk1 sshd(pam_unix)[4572]: 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=test.abc.net"
data_hard = "loc=544078|action=encrypt|i/f_dir=inbound|i/f_name=eth4c0|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={xxxx-6123-40CA-XXXX-9620355xxxxx};date=1190818929;policy_name=NYC-BellSouth NY]|src=10.100.0.50|dst=10.104.0.21|proto=icmp|rule=9|scheme:=IKE"


Delimiter based KV extraction as part of kv/extract command
OK, great! Now that you know what delimiter based KV extraction is and how to find the list of characters that are used as pair delimiters (pairdelim) and key-value delimiters (kvdelim), let's look at how to instruct splunk to perform this type of KV extraction. Well, all you need to do, is add the delimiters as arguments to kv/extract, as follows:

..... | kv pairdelim="?&" kvdelim="=" | .....
or
..... | extract pairdelim="?&" kvdelim="=" | .....


Configuration for automated delimiter based KV-extraction
transforms.conf is the key-value extraction configuration file. Delimiter-based KV extraction adds another configuration variable to the transforms.conf vocabulary called DELIMS - yes you guessed right this is where we'll specify the pairdelim and the kvdelim. The format of DELIMS is as follows:


DELIMS = <pairdelim>, <kvdelim>

Example:

in ...bundles/local/transforms.conf
.....
# this is equivalent to ..|kv pairdelim="?&" kvdelim="=" |...
[my_extraction]
DELIMS = "?&", "="
.....


You can then use the newly created transform just like any other transform. To remind the forgetful, you can do:
1. ..... | kv my_extraction |....
2. Automatically run"my_extraction" based on source/sourcetype/host with REPORT-* config variable in props.conf

I suspect others may be having this problem, as the problem lives in some very popular open source code as far as I can tell. With that, I'll begin telling you about my journey into hell.

Splunk has a home grown embedded HTTP(S) server that serves up all external interfaces to the 'splunkd' daemon. We use it as the core engine for our REST and XML/RPC-like API's. The GUI and the CLI both end up talking to the daemon via this server.

When I wrote the core of it a few months ago, I ran some rudimentary performance tests on several platforms and it seemed decent enough for our use, but a week ago, the manager of the Search and Indexing team (Stephen) said that he was seeing abysmal performance using SSL. He said that the GUI performance was being impacted. I didn't believe him and insisted that it was something else and that he was high.

So to prove to him that it wasn't my server, or my problem like all engineers do, I gave him a small python script that hits the server in a tight loop and we checked the performance. It sucked. Continuing with the theme of "this isn't my problem" - I told him it was probably the handler of the request that was doing something that made the server seem slow. This is when he laughed at me and said "watch this": He proceeds to turn off SSL, re-run the same test and the performance of the server goes up by approximately 50X. 50 times faster! I know that SSL is slower than non-encrypted streams, but there was no way this was the problem. Whoa! We can't ship this way. This needs to be fixed!

In fact, a very small HTTP request (approx. 80 byte) with a small reply (approx. 300 bytes) was operating at only 23 requests/sec! When he turned off SSL, he was getting over 1000 req/sec! What???

So, of course I tried the same test on my OSX laptop and I got 130+ req/sec - within the realm of reasonable and certainly better than 24. I then tried running the server on my laptop and the client on my Linux Fedora machine resulting in basically the same performance. Why does this work on my hardware and not his?

Finally, I switched the server and client by putting the server on my Linux box and the client on my Mac. I re-ran the test and damned if the performance didn't completely suck! I was getting 20 or so request-replies per second over SSL.

But, why does the OS matter? I didn't get it.

My SSL Performance Bug Diary

• Broke out ssldump. Here is a snippet from an OSX client and a Linux server. Note the third C-&gt;S line of .0398 seconds. This is the cause of the slowdown, but why?

• Spent 2 hours looking over every possible OpenSSL build option and try turning various ones on and off. No difference. (score: Bug 1, Rob 0)
• Spend many hours trying different crypto combinations. Little difference beyond the obvious and documented performance differences. (score Bug 2, Rob 0)
• Perhaps I need to throw in server-side SSL caching. I throw it in, with the assumption that the python client implements client-side SSL caching. No performance change. (score: Bug 3, Rob 0)
• Thinking it might be the Nagle algorithm, I modify my test to send larger requests and guess what? The performance is normal again! I try to find out exactly when it turns from slow to fast (as far as the request size) by trying request sizes of 1, 2, 4, 8, 16, 32........16K bytes. Wow, just around 1300-1400 bytes is where the performance goes from sucks to fast. Look at the graph below. See the spike? Hmmm..... (score: Bug 3, Rob 1)

• I change the MTU on the server from the default of 1500 bytes to 1000 bytes. The performance cliff now is lowered to somewhere in the 800-900 byte range. The MTU is the key! (score: Bug 3, Rob 2)
• It's got to be the Nagle algorithm. I try turning off the Nagle algorithm on the server. No performance change. (score: Bug 5, Rob 2)
• I give the problem to our performance engineer. He can reproduce it. I suck.
• Decide to try ssldump again and this time try a different test - curl sending the same size request as in the python test. I want to compare timings. BINGO. It's not the server, it's a combination of the server running on Linux and Python. (score: Bug 5, Rob 3). Notice in the following curl ssldump image, the single C-&gt;S line and the fast .0007 second timing. Contrast this to the previous ssldump image and here enlies the problem :

• Now to fix it. It really really seems like Python is the problem. I try it with urllib2. Same thing.
• I try it with httplib2. Same thing.
• I look at the code for urllib2 and httplib2 and guess what? They both use httplib. The problem must be in httplib. I dig into the code and start commenting shit out and looking at the resulting ssldump output to figure out *exactly* which write is causing the damage. I find the bug. (score: Rob wins)

The Problem and the Fix

I forgot to tell you that we are using Python 2.5. It turns out that httplib.py sends requests over the wire in 2 chunks. The first chunk is comprised of the HTTP headers. The second chunk is the body. The fix I made appends the body to the headers and sends the request in 1 chunk only. This is what curl does and this fixes the performance problems.

Here is the fix for download:

httplib.py

Here is my final data:

Things I Still don't Understand

Because it seems to work and this took so damn long, I am not going to do any further investigations, but there are still many unsolved mysteries. Perhaps one of you can figure them out.

• Why the extreme falloff on linux where both the client and server are on the same machine at 16K request/reply size?
• Why is OSX so much slower than linux?
• Why does the new code speed up linux only?
• Notice that only the OSX box gets the speed up at the MTU, the Linux box continues the slow performance regardless of the MTU

Windows to Linux Performance Numbers (added 2/5/08)

So I added a Windows to Linux graph based on the first comment I received below. Yes, we do test with Windows, and yes, it is not out yet (but will be soon). The problem manifests itself exactly like it does on other platforms. Notice the difference:

Specs on the Test Hardware

• Windows
  • Dual Core, very fast, lots of Ram (will provide detailed specs in a bit)
• Linux:
  • 2.6.11-1.1369_FC4smp
  • 3.4Ghz P4, Hyperthreaded, 2G Ram
• OSX
  • Mac Pro Laptop
  • 1.8Ghz Pentium Core II duo (2 cores), 3G Ram

It is this developer-centric ethos that sets us apart from so many of the other enterprise software firms and has already paid dividends on community goodwill. Instead of making prospective buyers jump through registration hoops just to view a guided webcast tour, Splunk provides fully functional software downloads to try out on your own data, inside your own network, free from webinar smoke and mirrors.

We don't just want you to try out the software, we want you to try doing things that aren't covered in our brochureware, things that sound ludicrous at first but are doable. In fact, in a perverse way, we hope that you do break our product because it reveals new limitations for us to solve, ultimately leading to a product that lets you do your job the way you want, yet easier and faster.

This is where the Splunk Platform comes into play. We want to increase the ubiquity of Splunk by, 1) exposing major components of Splunk as individual services, and, 2) allowing external developers to build on top of Splunk and leverage our award-winning IT search infrastructure. Starting with version 3.2 (you can download the preview version today), there is a new REST API that provides unprecedented access and consistency to every aspect of the Splunk Server. We are leveraging open standards like the Atom Protocol and OpenID to let enterprise developers create mashups with the same ease as those in the "web2.0&#8243; world. For programmers who want to integrate Splunk functionality into existing applications, you can look forward to Python and .NET SDKs in the near future, with Java and Perl not too far behind.

Amazon's Web Services, Facebook's F8, and Twitter's API have all proven that standardized platforms breed diverse applications, on scales that are much bigger than a single company can produce. That's the kind of ecosystem we want to cultivate. My next posts will begin exploring the new REST endpoints that have been added to Splunk, and provide tutorials on how to use those endpoints to interact with Splunk programmatically.

To play with a live instance, go to bash.splunklabs.com, login: guest, password: guest.

Of course, Splunk duplicates the functionality of the site itself. We can find, for example, the top 100 IRC quotes:

Splunk lets us do considerably more, though. What are the top one-liners?

How many more quotes mention "girlfriend" than "boyfriend", i.e. exactly how bad is this sausage party?

Are there any commonly quoted individuals?

Are there any interesting trends in quote scores over time? Take a look at high quote scores vs. quote ID:

It seems likely that older quotes, especially good ones, benefit from a disproportionately greater number of views (the rich getting richer, so to speak); this might explain why the peaks in the low-quote-ID ranges are higher than the peaks for more recent quotes. Or maybe the internet just doesn't produce the same quality of LOLs that it once did.

To try this yourself, add the following to props.conf:

[sourcetype::bash]
BREAK_ONLY_BEFORE = (#[0-9]* \+)|([0-9]+-[0-9]+-[0-9]+-[0-9]+-[0-9]+-[0-9]+)
REPORT-bash = bash

and the following to transforms.conf:

[bash]
REGEX = #([0-9]+) \+\((-?[0-9]+)\)- \[X\]
FORMAT = $0 bash_quote_id::$1 bash_quote_score::$2

Then, get a static copy of bash.org. You can grab the one I've created here, or you can generate it yourself:

$ curl -o '#1.html' 'http://bash.org/?browse&p=[001-409]'
$ for cur in * ; do lynx -dump -nonumbers ./$cur >> /tmp/bash.txt ; done

Finally, push the data into Splunk:

$ splunk add tail -source /tmp/bash.txt -sourcetype bash

I'm the manager of the search and indexing team at Splunk. We're still in the process of writing up our findings from storage benchmarks but here are the general details.

High IO/s typically means both faster indexing in general and faster searching of rare, temporally incoherent events. On average, we've seen indexing speeds increase by about 66% going from an 7200 RPM SATA RAID to a 15K RPM SCSI RAID. We've seen comparable performance from SCSI and SAS RAIDs, provided they're 15K RPM.

The best best benchmarking tool we've found for measuring how Splunk will behave on your disk hardware is bonnie++. If your disk subsystem can sustain 800 IO/s, you're in good shape.

As far as searching goes, IO/s is the dominant factor for non-coherent, infrequently accessed search results. This means, if you're just searching for the newest data, or even have to reach back through 1MM events to return 10k, the disk is NOT the bottleneck, since each individual read() will pull many events off disk. However, if you're searching for a rare term, like a name, that occurs once an hour or once a day, each read() is going to require the drive arm move. If you're using a 7200 RPM SATA drive, that's about 100 IO/s and hence on the order of 100 retrieved events per second. If you have a decent RAID, that could be 800 retrieved events per second.

-s

It's taken longer than we would have liked but our 3rd preview build has been posted.
Get'um here

A bunch of work has gone into windows stability, tons of bugs were fixed, and a bunch of customer requests have been implemented ( we will let you know out of band ). We expect that this release should be more stable, slightly faster, and less buggy.

Left to do, we still have a bunch of IE work, performance improvements, and cleaning up of some features like interactive field extraction and event type discovery.

Its still not production ready so don't even think of trying it out for real - and there is no guarantee that migration will work from a preview to GA ( we will migrate from 3.1.x to GA but not preview ). Also, don't run splunk as root - its just not good to do until we run through all our testing.

As always, please send us feedback at splunkpreview@splunk.com or hit us up on IRC (irc.efnet.org #splunk).
The last round of info from Preview #2 was awesome please keep it up!

e.

Two things struck me interesting about Charlie's post.

First, he noticed the changes in the UI we've been slowly making over the last few releases. If you've ever done UI design, you know how much sweat goes into every little detail, and how much momentum a design carries over time. That someone noticed the new changes *and* liked them is a HUGE win for the UI team. It's even better how fast someone noticed!

Second, he actually spends quite a bit of time explaining the security workaround in the free product - one that I covered earlier, coincidently enough. I figure if someone goes to the time and trouble to figure out how they can keep using the product in a secure, legitimate way, then they must really, really like it. You simply can't argue with an evangelist like this.

If anyone here is a gem, it's Charlie.

• From the commandline: "splunk ftw" produces an ascii-art "O'Rly?".
• From the commandline: the "outputrawr" produces ascii-art fireworks.
• From the searchbox, piping results to the "marklar" processor (e.g. "*|marklar"), converts all search results into the Marklarian language.
• From the searchbox, piping result to the "loglady" processor (e.g., "*|loglady"), converts all the search results into quotes from Twin Peaks's LogLady.

Enjoy them while they last, before they are removed by the Silliness Police, who%$($%%$
^H^H^H^NO CARRIER

Problem definition:
Extract structured information (in the form of key/field=value form) from un/semi-structured log data.
Note: for the purpose of this post key or field are used interchangeably to denote a variable name.

Problem examples:
Splunk debug message (humans: easy, machine: easy)

12-03-2007 13:51:55.114 DEBUG SearchPipelinePerformance - processor=save queryid=_1196718714_619358 executetime=0.014secs
ideal structured information to extract:
processor=save
queryid=_1196718714_619358
executetime=0.014secs

Splunk tries to make it easy for itself to parse it's own log files (in most cases)

Output of the ping command (humans: easy, machine: medium)

64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=2.522 ms
ideal structured information to extract:
bytes=64
from=192.168.1.1
icmp_seq=0
ttl=64
time=2.522 ms


An interesting pattern to note here is that there is no consistent field-value delimiter, nor field-value order. In the "from" field the authors have chosen to use a space as a delimiter, while for "icmp_seq", "ttl" and "time" they've chosen the equal sign. For the "bytes" field they've chosen to place it after the value (yes, they might have also intended for it to mean bytes - the data unit) while for the rest they've chosen field-name followed by field-value. Admittedly, some might think the current format is prettier than the following consistent log line which could easily be parsed by machines. (Who thought log files were optimized for prettiness !?)

bytes=64, from=192.168.1.1, icmp_seq=0, ttl=64, time=2.522 ms


NetScreen log (humans: medium, machine: hard)

%MD% %DD% 13:41:25 45.2.0.1 NOC-FWa: NetScreen device_id=NOC-FWa [Root]system-notification-00257(traffic): start_time="2006-05-11 13:40:23″ duration=62 policy_id=41 service=Network Time proto=17 src zone=noc-mgt dst zone=noc-svcs ......
ideal structured information to extract:
device_id=NOC-FWa
start_time=2006-05-11 13:40:23
duration=62
policy_id=41
service=Network Time
proto=17
src zone=noc-mgt
dst zone=noc-svcs


This part of the NetScreen log line ...service=Network Time proto=17 src zone=noc-mgt dst zone=noc... is a salient example of the ambiguity that sometimes exists in log data. What is the correct value of service ? "Network" or "Network Time"? What about the name of the next field? Is it "Time proto" or just "proto"? Well, we can come up with an easy rule for this case, let call it Rule-1: "Field names should NOT contain spaces". Fair/good enough!
Let's move on to the next field, what is it's correct name? "src zone" or just "zone"? A human can recognize that "src zone" is the correct field name, thus we just violated the our Rule-1, we can continue our cycle of adding/violating/modifying|removing rules to our rule set only to recognize that the cycle never ends - which simply translates into "there is no one solution/rule-set that is able to extract structure from ALL unstructured data" - there will always be a degenerate case that violates the rules.

More degenerate log lines:
Stay tuned! Links in this section are coming soon....

Solutions:
- Delimiter based key-value pair extraction
- Delimiter base KV extraction - advanced
Stay tuned! More links coming soon....

Here is a little technique for indexing JavaScript exceptions in your production and development environments using Splunk.

In JavaScript create an onerror event handler that makes an HTTP request to a server that has access logs indexed by Splunk.

    function JSErrorLogger(httpBeacon){
        var self = this;
        self.handler = function(msg, url, line){
            var log = {
                "date":new Date(),
                "type":"jserror",
                "line":line,
                "msg":msg,
                "url":url
            }
            var logStr = "";
            for(var i in log){
                logStr += i + ":" + log[i] + " ";
            }
            var imgObj = new Image();
            imgObj.src = httpBeacon + "?" + logStr;
        };
        self.JSErrorLogger = function(){
            window.onerror = self.handler;
        }();
    }

Make sure that this JavaScript is the very first item executed by the interpreter, ensuring all exceptions are caught by the event handler.

Instantiate the class with a URI that points to a beacon on a machine that has Splunk indexing the access log. You may want to set some environment variables in JavaScript that turn logging on for only testing and production machines.

   //if environment test or production
   var splunkJSErrorIndexer = new JSErrorLogger("http://somedomain.com/beacon.gif");

That's it, now you can empirically understand JavaScript exceptions being raised, set blackberry alerts and correlate ui stability issues to deploys:)

Happy JavaScript Monitoring!

What you really want to do is refer to them with different subdomain names, where something like http://splunkpreview.mydomain.com/ would bring up Splunk without having to remember the port number.

If you are running Apache, (like I am on Leopard) you get a reverse proxy server for free. With just a few lines of configuration, you can alias subdomains (or domains for that matter) to your heart's content.

You also get the ability of putting content behind some basic authentication provided via Apache's HTTP auth methods. This comes in handy if you'd like to link to your Splunk install from a publicly facing page, but don't want people to know what type of content is behind the authentication. It also works for limiting access to a particular IP address group or domain.

I've put together a screencast covering how to do this from OS X's version of Apache. Click on the thumbnail below to play the screencast.

Note: Firewalling the actual port Splunk runs on is left as an exercise for the viewer, as is limiting access to a group of IP addresses. More information about configuring Apache's mod_proxy module can be found on Apache's website.

Here's the configuration code from the screencast:

Its packed with holiday goodness, albeit very raw.

First you will notice we have posted a windows build. Its been in the cooker since last Feb and thanks to Mitch, Ledio, Igor and a bit of Amrit we now have a single code base that rocks on linux, mac, solaris, freebsd, aix, AND windows. This was not an easy feat as evidenced by our gift of a pony (soft and electronic) to Mitch for his effort. Its still very raw (the build not the pony), and has a tendency to crash because of a memory fragentation and limited vm space. Which will be fixed by GA... MarkB. will post more on the build so stay tuned for details. Its a big deal for us so be patient and we sure could use feedback on how to make it the best it can be.

Also in this release you will see the UI starts to get some of the async search results. Over the next few releases we will be moving to fully async search in the UI. It will take a few turns but this preview has some of the first cut.

There are a bunch of other improvements; scheduled searches got a bit of a cleanup in the UI and the backend has been improved as well. Performance, bugs, and other tweaks are also spread throughout. I'll get others to post specifics.

In the mean time, as always its a huge help to us in dev if you can kick the tires before we freeze for GA. Please send feedback to splunkpreview@splunk.com, post comments to this blog, or drop by and tell us in person.

Again, thanks for the help and happy new year from all of us in dev@splunk!

(Update): While watching the video again and realized I sent a mixed message about where to edit configuration in splunk. I made it clear that you want to edit in the local bundle directory, and if you look at the terminal that is where I was editing my configuration, however, I later said "default over-rides local, so always edit in default", this is WRONG. Always make your personalized configuration changes in the local directory, if the configuration file doesn't exist there, create one or copy it from default and edit that one.

Take a look at the video and let me know if you have any questions about this stuff.

Quicktime Video (625&#215;352)

More information can be found in these videos:

• A quick walkthrough of the new preview release feature Live Tail, its UI, and some sample code - See Video
• An overview of the architecture used to integrate real-time data from Splunk Live Tail in a web browser. Challenges and workarounds when using JavaScript/Flash hybrids - See Video

Happy Streams!

So we have added in flexible roles into the preview release. Well, what does that mean.
We will now allow folks to create their own roles. The previous ones of Admin, Power
and User will be included as defaults.

There is currently no GUI available for editing roles but you can directly edit the
config file $SPLUNK_HOME/etc/bundles/default/authorize.conf.

To add in these roles we did an audit of our system and broke down various actions
into capabilities. These capabilities can be grouped together to create any role.
Please bear with us here, this is just a first cut and we may not have chopped up
things in a way that makes sense to you. This is the beauty of preview, you got a suggestion
about capabilities you'd like to see added or removed then comment or mail us.
The more feedback we get at this stage the faster this feature will improve.

A role in the splunk system contains the following things.
1. A list of capabilities that role can perform.
2. A list of roles that are contained within this role ( their capabilities will be imported into our role)
3. A list of search filters that should be applied when searching as this role.

Below demonstrates how to define a role called kwyjibo that can edit users information and
make changes to the authentication system. It imports in the capabilities of the roles User and Power.

[role_kwyjibo]
edit_user = enabled
change_authentication = enabled
bounce_authentication = enabled
importRoles = Power;User
srchFilter =

If you have any questions, comments please let me know.

Rory

We have posted the our first of many preview releases. You can find them here:
Our hope is that every week or two as new features or API's become usable that we post builds soliciting feedback.
This first post has a bunch of backend and UI performance improvements as well as some new but hidden features:

• live searching of data
• flexible roles
• scripted authentication
• event decoration ( for the xmas season )
• auditing of splunk server actions
• file system change detection
• improved (proper) sub second support
• transaction search
• new experimental simple search interface
• "where" support in search clause ( you dont need to use the "| where" anymore and can just search for foo=10 )

I'm not going to explain here what these things mean or how to find them or use them
Instead the product managers and developers will post here with ideas on what to try and what feedback we are looking for.

I'd like to thank in advance those brave few of you that have the few minutes to install these builds and give us your feedback.

e.

Just a heads up that we are moving to a model where we post previews of upcoming releases.

Starting now, we are going into a mode where long before a GA release we will be posting development builds. At first, they may be a few weeks apart but over time our goal is to post builds as soon as new functionality or API's are ready for comment.

This first Preview #1 will have backend performance and scale improvements as well as some cool new features. The developers and PM's will be posting to this blog the specifics of what is new, how to try it, and where we are going.

Our hope is that we get early feedback on new features and API's before we actually ship.

Thanks in advance for helping try out our early wares.

Kinds Regards,

e.

If I have a search/report that I want to run faster, I will save that search and have splunk run it over a small timeframe (5,15,30,60 min) taking the results of that search/report and feeding them back into an index i create to hold cached results.

For example, suppose I like to run nightly reports where I show "top users by bandwidth". Its easy enough to run the report every night, but suppose there are times during the day when I want incrementals, or I want to look at last week, or perhaps get dailies over a month. Every time I run the search/report I need to search and recalculate "top users by bandwidth", which if over billions of events can take time

Instead, I'll just save the search/report and have Splunk run it every 15 minutes with the results being sent to a "cache" index. This way if I ever want to do an adhoc search on "top users" or if I want to do "weekly reports by day" all the data is precalculated.

Think of this as creating "logs" that are the output of a search/report and then having Splunk index those "logs". To get fast results you can then search/report on the summarized cached data.

If not obvious why it's faster, suppose you are indexing 500M events a day and 100M of those have bandwidth data. To report on "top bandwidth by users" I need to run a search to get the 100M events then run the report across all 100M.
If instead I were in the background running that same search/report over each hour interval, then saving the data back into splunk, I would reduce the data i'm operating on from 100M down to 1200 ( 24*500 ) (assuming that i'm getting top 500). Doing searches/reports on the later dataset are sub second versus the few minutes it would take to run across the 100M.

Make sense ? - its really simple but odd to explain.

PART ONE - Setup:

• 1. Grab the reportcache search script from "** here ** and put it in your SPLUNK_HOME/etc/searchscripts directory - no need to restart you can now cache any search/report data.
• 2. Add a cache index - either add the following to your etc/bundles/local/indexes.conf or create a new bundle and add to that indexes.conf You will need to restart splunk after adding the index.
  
[cache]
homePath = $SPLUNK_DB/cache/db
coldPath = $SPLUNK_DB/cache/colddb
thawedPath = $SPLUNK_DB/cache/thaweddb


PART TWO - Testing by writing to a file:

I recommend that you first test reportcache by having it output to a file that you scan to make sure things look right.

• 1. Find a search you want to cache. Simple candidate is something like the following report against the internal index that shows queue sizes by queue name.
  index=_internal metrics "group=queue" timechart avg(current_size) by name
• 2. Once you have a search you want to cache - add the following "reportcache index=cache path=/tmp file=testcache.log notimestamp" command to the end. The following assumes you have made an index named "cache". The index attribute is required and you should not use your default unless you know what your doing. Also we are going to output the file to /tmp/testcache.log using the file and path attributes. The notimestamp option simply suppresses adding a timestamp to the filename.
  index=_internal metrics "group=queue" | timechart avg(current_size) by name | reportcache index=cache path=/tmp file=testcache.log notimestamp
• 3. Run the search and you should get back the normal search results and not see an error on the screen. If you do see an error it should be self explanatory.
• 4. Open the file /tmp/testcache.log and make sure the results look ok. They should look like a bunch of lines key=value, key=value

PART THREE - Writing to an index:

• 1. We are now going to have the command put the results into the index. Simply remove the file, path and notimestamp attributes
  index=_internal metrics "group=queue" | timechart avg(current_size) by name | reportcache index=cache
• 2. Run the command - you should again see normal results and no error.
• 4. Wait 30 seconds or so...
• 5. Run the following search to make sure results made it into the cache index - you should see your cache data after this search
  index=cache
• 6. Now click on the report link and see if you can get your report back This part is the somewhat odd part. All the fields should be as they were in the original search but many reports create keys with odd names. The best thing to do is to click around and see what reports you can make. You should be able to get back to the original search/report prior to the caching.

PART FOUR - Enabling automatic caching:

After you have found and tested a search/report you want to cache moving forward:

• 1. Save the search along with the reportcache command
• 2. Schedule the saved search on a small time frame ( 5, 15, 30, etc ) minutes
• 3. Test by waiting a few hours and looking at the results in the cache index.

There is a good chance that either the above description was vague or that there is a bug / edge-case that i did not consider.
One frequent problem i have seen is trying to cache data that has no timestamp. For example,
somesearch | top users
will produce restults without timestamps. This makes a mess of the cached data. If you have this problem then try rewriting your search to something like:
somesearch | stats count first(_time) by users | where users != "" | sort -count
The above will produce data that has both top and timestamps.

Few other things that are common requests:
Often folks want to go back in time and create cached results for prior data. I have a script that can do that and will post it after more testing.
Another common topic of conversation surrounds the over creation of summary data. In many cases it can be benificial to cache more stuff than you initally need in case you want to run reports later. I'm trying to think of good ways to automatically do this for you.

** IMPORTANT ** - drop me a line and let me know how something like this *should* work. I suspect that we will add a "checkbox" to saved searches that will automatically do the right thing.

I'll leave this post wit the usage info from the top of the search script.

# usage: | reportcache
#
# file=[filename] - default is current time
# path=[path] - default is $SPLUNK_HOME/var/spool/splunk
# index=[indexname] - which index to target for results. If blank will use whatever is bundled
# marker=[string] - this is just a token or k=v used to mark the results for version or other delination or to defeat crc caching
# format=["csv"|"splunk"] - use the output format "splunk" for feedking back into splunk or csv if you want to save for other tool
# appendtime - if true this will append current time. Its useful when you are doing something that you want with a timestamp of now
# notimestampe - is this arg is supplied it will suppress the timestamp in the filename
# debug - if debug then will just out args to screen

# following example will put in var/spool/splunk a file named foo without timestamp, marked with erik=nextrun", and targed to index cache
# index::_internal "group=pipeline" | timechart avg(executes) | cacher file="foo" notimestamp marker="erik=nextrun" index="cache"

How are things? so I've made some progress in my attempt to code myself out of a job. Just checked the scripted auth into the preview branch which should be released in a few days. It's very basic right now with more improvements to come. At the moment userLogin, getUserType and getUserInfo are the only methods you need to fill in.

I've written up a sample that interfaces with PAM on the linux, using /etc/passwd to get user lists. Mac users skip the pamauth.c compile you don't need this app and pam don't like macs ( can't say I blame pam on that score)

First off a pamauth.c program to compile that will talk to pam for ya. Donated by Phillppe Troin, thank you fif. Feel free to take and edit for your own purposes, but you must send fif a chocolate chip cookie if you found it useful.

File pamauth.c is attach due to severe lameness on part of wordpress, insisting on screwing with the #include's

pamauth.c

Compile that puppy like so
gcc -Wall -Wextra -o pamauth pamauth.c -lpam

You may need to create an entry for pam
edit /etc/pam.d/pamauth and put this line in
auth sufficient pam_unix.so

To access pam root access is usually required so we will just set the pamauth script setuid instead of running splunk as root (which would be deeply stupid BTW).

as root:
chown root.root pamauth; chmod a+s pamauth

You can test it by doing echo PASSWORD | ./pamauth username
returns 0 for auth passed
returns 1 on fail.

K now that you have your nifty pam app running you need to add your python script that will interface
with splunk. As they say on cooking shows, here's one we made earlier.

[source:py]
# Required functions;
# 1. userLogin : login with username password pair
# 2. getUserInfo : get user information. passed back in the form.userId;username;password;realname;userType
# 3. getUserType : the splunk role to attach that user to.
# optional functions
# 1. getUsers : Enumerate all users in the system, these will then be displayed on the user page in splunk.
# Later release
# 1. checkSession : Current version just auths and then splunk managed the session, this will allow
# session management to be handled here. Careful though splunkd and the frontend
# are quite chatty this will be called alot. If it's slow it will degrade performance.

import sys
import subprocess

SUCCESS = "success"
FAILED = "fail"

PAM_EXE = ""

def writeToStdout( listIn ):
result = ""
for fu in listIn:
result = result + "[" + fu + "]"

sys.stdout.write( result )

def readFromStdin( ):
input = sys.stdin

inStr = ""
for line in input:
inStr = inStr + line

inStr = inStr.replace( "[", "" )
return inStr.split( ']' )

def userLogin( infoIn ):
listFu = []
username = infoIn[0]
password = infoIn[1]

command = PAM_EXE + infoIn[0]

# our check with pam is done with a setuid program called pamauth
proc = subprocess.Popen( PAM_EXE + ' %s' % username,
shell=True,
stdin=subprocess.PIPE,
)
proc.communicate( password)
retCode = proc.wait()

if retCode == 0:
listFu.append( SUCCESS )
else:
listFu.append( FAILED )

return listFu

def getUsers( infoIn ):
listFu = []
listFu.append( SUCCESS )
# just going to use /etc/passwd here but you may use any method you wish.
FILE = open("/etc/passwd" ,"r")
fileLines = FILE.readlines()

for line in fileLines:
userBits = line.split( ":" )
if userBits[6].find( '/bin/bash' ) != -1:
realname = userBits[4]
if realname == "" :
realname = userBits[0]
# userId username password realName userType/splunk role
listFu.append( userBits[2] + ";" +userBits[0] + ";***********;" + realname + ";Admin" )

FILE.close()

return listFu

# IN UserId
# OUT [RESULT(SUCCESS|FAILED)][userType]
def getUserType( infoIn ):
# Here you are given a userId
# you must return the user type (splunk role)
# I'm just going to make everyone an admin.
listFu = []
listFu.append( SUCCESS )
listFu.append( "Admin" )
return listFu

def getUserInfo( infoIn ):
listFu = []
listFu.append( SUCCESS )
#userId;
listFu.append( infoIn[0] + ";" + infoIn[0] + ";***********;" + infoIn[0] + ";Admin" )
return listFu

if __name__ == "__main__":
callName = sys.argv[1]
listIn = []
listIn = readFromStdin( )

returnList = []
if callName == "userLogin":
returnList = userLogin( listIn )
elif callName == "checkSession":
returnList = checkSession( listIn )
elif callName == "getUsers":
returnList = getUsers( listIn )
elif callName == "getUserType":
returnList = getUserType( listIn )
elif callName == "getUserInfo":
returnList = getUserInfo( listIn )
else:
returnList.append("ERROR call name no known" )
returnList.append( callName )

writeToStdout( returnList )
[/source]

Change the PAM_EXE variable in the script to point to the app that will check the password. On linux : the pamauth module you just compiled. On Mac (the piano-accordion of computers): use chkpasswd program shipped with mac.

Now that you have a script auth plugin ready to go all you need to do now is tell splunk about it.

Example of the authentication.conf bundle.

[source]
[auth]
authSettings = fubar
authType = Scripted

[fubar]
programPath = /opt/splunk/bin/python
scriptPath = /home/boo/splunk/scriptedAuth/flubber.py # my python auth script.
[/source]

Now pay attention here you do need to edit programPath and scriptPath to paths on your system.

Things left to do.
1. Allow users to pass back search filters on userLogin and getUserType.
2. Allow session management to be handled by scripted input. ( right not once auth is confirmed as correct splunk takes over session management).

Also this script will not return user lists on the mac ( not big deal you just can't see all users in the admin/users tab ). Erik Swan has volunteered to fix this because he loves macs, a little too much really it's kinda unhealthy.

Download this and play with it, let me know of any problems.

I will publish more details on the communication between splunkd and the script but for the moment you folks can reverse engineer this, it's pretty simple, a lame wilder beast could figure it out.

More later, for now it's time for beer pong, played for cold hard cash and ugly women.

Ciao,
Rory

Unfortunately, this week we ran into a rather tricky memory leak when using this nifty class. An event listener was subscribed to the progress event and over time memory usage steadily increased to a point of making the browser inoperable.

After a little digging we narrowed the problem down to the URLStreams usage of the ByteArray. It seems as if URLStream was reallocating a buffer for the array and the short turn around time (on the reads) was not giving the garbage collector enough time to throw out the old allocation.

The way the leak could be corrected was by deleting the ByteArray (Set null), forcing garbage collection of the read buffer.

Here is the workaround:


var bytes:ByteArray = new ByteArray();
this.readBytes(bytes, 0, this.bytesAvailable);
bytes = null;


Happy Streams!

So anyway, I bring it up cause Johnvey saw one of it's cousins out in the wild, taking the whole concept to an extreme. Check it out.

Arguably though, this is so extreme that it's not reallyDescriptiveNames at all, but closer kin to a sort of passiveAggressiveWorkplaceSabotageAdapter.

Good. Then that means that we at Splunk are obligated to play play beer pong every Friday! I figure that with all the bottles and cans that subsequently go into the recycling bin, we're probably offsetting a small percentage of the many computers we use here... amirite?

If you disagree, you can voice your opinions in person. See you here Friday at 5PM.

I mentioned this in passing to one Sean Dick who is a developer friend of mine in Oklahoma City. What follows is a nearly identical post to the one he made over at his self-named blogpost on Blogger on how to get Rails to integrate with Splunk. "There's plenty left to do.", he said, but I'm convinced it's worthy of mentioning here. Thanks for hammering this out Sean!

Serious Material from Sean Begins Here

As per the norm, this post assumes you've downloaded Splunk for your particular platform. It also requires a newer install of Ruby on Rails. Come back when you've completed both these tasks.

Get Splunk started now:

> sudo export SPLUNK_HOME=/opt/splunk/
> sudo ./opt/splunk/bin/splunk start


You need to drop the Splunk on Rails plugin into your Rails app's lib folder and then call it with require 'splunkbase' Note: If you're running Splunk remote or on a non-standard port, don't forget to change the SERVER variable in the plugin file!

Now let's say you want to make sure your rails application is running bug-free, and when one does pop up, you need to know it pronto. You'll create a new controller into which we're going to put some splunk goodies. I named mine SplunkController, but you can be more creative.


class SplunkController < ApplicationController
require 'splunkbase'
@@foo = SplunkBase.new
def index
end
def reports
@document =" @@foo.splunkSearch('q' => params[:query])
end
end


This is really nothing more than making available the response from splunk to your view in the form of a variable. Defining a page for it to be displayed in is no more difficult than:

<pre><%= @document %> </pre>


Now we build the index page we defined so we can pass it the query:

<html>
<head>
<%= javascript_include_tag "prototype" %>
</head>
<body >
<%= form_remote_tag(:update => "graphDiv",
:url => {:action => :reports }) %>
<%= text_field_tag :query, nil, {:size => "100"} %>

<%= submit_tag "Get a report on your query" %>
<%= end_form_tag %>
<div id="graphDiv">
</div>
</body>
</html>


And believe it or not we're ready to start asking Splunk some questions. Try giving it something like:

[search sourcetype::what_you_named_your_source error starthoursago=24] | outputxml


As you've probably gathered, that'll give you a formatted list of all of the errors that have occurred in the last day.

Now let's say you've got a rails application running internally that you don't have the option to/don't feel comfortable with outsource analysis to something like Google Analytics. Back to our cute little controller, we add in a new definition for the graphing page.

class SplunkController < ApplicationController
require 'splunkbase' #those two magical words
@@foo = SplunkBase.new
def index
end
def reports
@document = @@foo.splunkSearch('q' => params[:query])
end
def graph #for graphing, this fixes things up so we can display the data
@datahash = {} @queryDoc= @@foo.splunkSearch('q' => params[:query]) #here is the meat
@queryDoc.each_element("//r/") do |ele| #here we're sorting out what is useful
@datahash[ele.elements["m[@col='1']"].text] = ele.elements["m[@col='2']"].text.to_i
end
@sorted = @datahash.values.sort.reverse #sorting it for the hell of it
@chartheight = @datahash.values.max + 50 #to make it look pretty and consistent
end
end


This example is fairly simple and assumes you're just looking for basic metrics on your site's usage. You could build it larger to accept whatever you want splunk to throw back at you. This one expects to see something like "Content Name" => "value".

Now let's take a stab at setting up the graph:


<samp><%= @queryDoc.to_s %></samp> #gives us a raw return of the data we pulled from Splunk
</div>
<% @sorted.each do |name, height| %> #Iterate through each of the data pairs and grab the height.
<div class="columnSpacer">
<div style="margin-top: <%= 100 - ((height * 100)/@chartheight.to_f) %>%"class="graphTitle">
<%= name %>
<br>
<%= height %> hits
</div>
<div class="graphColumn" style="height: <%= (height * 100)/@chartheight.to_f)%>%">

</div>
<% end %>


In the interest of keeping things from getting too esoteric I've committed a no-no and left some programming in the view. All in all, it's pretty light math to get things displaying properly. As you should be able to glean from the code presented we're just iterating through each of the name/value pairs we extracted from the XML Splunk returned and turning them into pretty little bars on a chart. Now all we need to do is put together the index page for accessing the graph function.


<html>
<head>
<%= javascript_include_tag "prototype" %>
</head>
<body >
<%= form_remote_tag(:update => "graphDiv",
:url => {:action => :graph }) %>
<%= text_field_tag :query, nil, {:size => "100"} %>

<%= submit_tag "Get a report on your query" %>
<%= end_form_tag %>
<div id="graphDiv">
</div>
</body>
</html>


There's pretty minimal monkey business here, so let's go on to the fun part:

Let's take a look at what controllers are getting the most face-time for our users and what content sections are being perceived as being the most useful. This example comes from a site I did recently for a client and happens to be the most handy rails logfile I have within reach.

Pop in the query:

[search sourcetype="the_name_you_gave_your_source" | top 5 controller ] | outputxml


And you get something like this:

There are a myriad of options available to you through Splunk's search interface, and learning to romance the queries to give you what you want would be a section all its own. This one, however consists of limiting the scope of the search ( sourcetype= ) and giving it a context to put it in ( top 5 controller ) - in this case, the top five controllers.

Next post I will cover the possibilities afforded with the use of bundles in Splunk in conjunction with your Rails application. In the meantime I highly suggest you peruse the REST API documentation supplied in your Splunk install and the admin/developer documentation on Splunk.com to get a more in-depth understanding of what you can do.

Below is a second screen capture video that I just made to describe Splunk's new Event Typer. The Event Typer dynamically tags system events in custom, yet, universal ways. For example, I can say that for any event that happens on Sunday, that has 'status=Fatal', and that has "sourcetype=weblogic", to be dynmaically tagged as a "weekend_fatal_weblogic" event. Topics covered include: what is an event type; how to search, view, and count event types; creating an event type; creating an event-type template; and discovering event-types.

Yes, production value is what you've come to expect from a Carasso Production. That's right 15 minutes of unscripted nerd talk. Now with a bonus 45 seconds of video as I type in an off-camera window. But I promise you'll learn a few useful things you didn't know.
EventTyperVideo (15 minutes of emacs magic)

Over time I've compiled a small library of scripts for various p4 functions that have been written time and again at different sites...mergetool is one of them. This little tool accepts a merge target ("yours" in p4-speak) and projectile ("theirs" in p4), labels both, performs an integrate, and performs a "safe" resolve -as. It logs any failures for you to resolve by hand, or submits the change set if the resolve completes successfully. It does this with a bunch of logging in a well-organized, date-stamped directory suitable for archiving (or splunking).

Yes, production value is at a new low here as I cover 18 minutes unscripted, but I promise you'll learn a few useful things you didn't know. There's a free Splunk t-shirt for the commentor that guesses the actual number of times I say "uhhhhh".

File ClassifierVideo (18 minutes of action packed emacs video)

Now this is going to sound pretty circa 98, but several main stream browser plugins support a JavaScript communication layer. According to the Millward Brown survey plugin installations of Flash (99%) and Java (85%) are pretty ubiquitous.

Flash/JavaScript Communication
The Flash ExternalInterface class enables communication between JavaScript and the Flash Player. ExternalInterface was first introduced in ActionScript 1.0; so Flash Player 8 is the minimum plugin version required.
From JavaScript

• Call an ActionScript function
• Pass arguments
• Return a value to the JavaScript callee

From ActionScript

• Call a JavaScript function
• Pass arguments
• Pass various data types (Boolean, Number, String, etc...)

Java Applet/JavaScript Communication
The scarcely documented LiveConnect API provides JavaScript with the ability to call methods of Java classes and vice-versa. Using LiveConnect in applets requires the mayscript attribute and the plugin.jar package for newer versions of Java (Howto for Mac OS X users). Communication from Java to JavaScript is mitigated through the netscape.javascript.JSObject class. JavaScript exceptions in Java can be handled using the netscape.javascript.JSException class. Public methods in an applet can be called using the applet container object followed by the method name and arguments (e.g., document.getElementById("myapplet").publicAppletMethod(arg1, argN);).

From JavaScript

• Call a Java method
• Pass arguments
• Return a value to the JavaScript callee

From Java

• Call a JavaScript function (Note: does not seem to support deep objects obj.foo(arg))
• Pass arguments
• Pass various data types (Boolean, Number, String, etc...)

It looks like LiveConnect is due for an overhaul in the near future, so you may want to keep your eyes out for changes on Mozilla developer Josh Aas's blog.

What's Next
With the power of Java and Flash this opens up the arena for creating visually hidden gateways (i.e., width:0px; height:0px; applets or swf movies) that extend the browser. Stay tuned for the next part in this series where we make a sample application. Feel the power!

Surprisingly, this is not an uncomfortable place to be. In 11 years in industry I've worked in a variety of organizations: the now-bankrupt dot-com best known for putting an ad with a naked guy up during the Super Bowl, 2 major marquee names with vastly differing corporate cultures, a security start-up stocked with emancipated-minor hackers. Aside from that doomed dot-com - which had a surprisingly strong gender balance throughout technical roles and a culture blessedly free of gender-based intimidation at all levels - Splunk may be the most comfortable place I've ever worked. There's no creepy tokenism (unlike stories I've heard about certain other bay area employers), That Guy Who's Never Seen A Girl Before doesn't work here...and as far as I can tell, no one really gets harassed except Amrit.

Perhaps a better testament for the dev culture than my opinion - because, frankly, I'm pretty weird to start with - is that other women in the company seem to be pretty comfortable visiting the dev area, either on work errands or just to take a break from the sales-focused environment upstairs. Frankly I can't imagine that happens too often in the bay area...and more's the pity.

Abstract
This paper presents an interactive bootstrapping process used in Splunk that automatically learns to extract fields from log events. End users simply select one or more example values of a field and a learning process discovers additional instances, along with the patterns to extract them. The user is able to correct the instances and save the extraction patterns. Immediately afterward, while searching log events the newly-taught fields will be extracted from the event's raw text.

Click here to read full paper

Feedback appreciated.

In this installment, I took it to the next level by using Splunk's search capability to overlay performance metrics on the diagram. The combination of Splunk logging metrics information for each processor within each pipeline (thanks Brad) and the ability to have Splunk execute a search processor written in Python made this possible. Here is how you use it:

First download graphviz. I particularly like the OSX application that they've written because you can see the graph on the screen and as the file changes, those changes are reflected in the graph you are viewing. If you don't have a Mac, use the command line version to generate different types of output file formats like .jpeg, etc.

Go to SplunkBase to download my python script. Copy the .py file into $SPLUNK_HOME/etc/searchscripts

Start Splunk.

Type the following into the search box:
This will search for the appropriate metrics information and pipe the results through the script.

There are 2 options to perfgraph:

perfgraph [output filename] [cpu, execs, cumhits]

Unfortunately (because I'm lazy) you can't specify cpu, execs or cumhits without also specifying an output file.The parameter is the full path and file name of the 'dot' file you wish to create. It defaults to /tmp/out.dot.

The second parameter, if specified tells the script to highlight in red the slowest processor (cpu), the processor with the most hits (execs) or the processor with the most cumulative hits (cumhits). This parameter defaults to 'none', or no highlighting.

The above search string results in the following graph (portion). Notice the performance information overlayed into the processors:

If you specify the output file and 'cpu', the processor with the most cpu time will be highlighted. Here is the search:

It results in the following graph (portion). Notice the red processor:

Next steps:

• Overlay queue metrics into the queue nodes
• Overlay indexer throughputs into the indexer nodes

You see. Splunk provides endless fun. Insane! Enjoy.

Let's get dirty. Go into settings..general..auto-lock and set locking to 'never'. This will keep the phone on while you hack around on it. Keeping the phone on and connected to the network will drain your battery like nobody's business, so make sure you plug in the charging cable.

Now install AppTap. Follow the instructions, and come back here when you are all done.

Using the AppTap installer on the phone, install the Community Sources, BSD Subsystem, Term-vt100, OpenSSH, Tinyproxy, and UIctl apps, in that order. UIctl will let you stop and start sshd on the phone. Launch it now to see if sshd is running. Click on the 'load' button if it's not.

Ping your phone from your computer with its IP address. You can use the terminal on the phone to grab the IP address:


# ifconfig en0
en0: flags=8863 mtu 1500
inet 10.0.1.194 netmask 0xffffff00 broadcast 10.0.1.255
ether 00:1c:b3:f0:0b:a6
#


Ssh to the phone from your terminal. The default root password is 'dottie'.


foobar:~ kord$ ssh root@10.0.1.194
root@10.0.1.194's password:
Last login: Wed Oct 10 13:45:22 2007 from 10.0.1.191
# hostname
Kord's iPhone
#


Now add a syslog.conf file to /etc/:


bash-3.2# echo "*.* @10.0.1.191" > /etc/syslog.conf
bash-3.2# cat /etc/syslog.conf
*.* @10.0.1.191


Obviously, you'll want to use the IP address of the machine on which you are going to install Splunk. Speaking of Splunk, at this point you should already have it installed. If you don't, download it here, and install it now. You can reference my first hack for instructions on getting Splunk up and running quickly on your system. Smile. Splunk goooood.

Back in your ssh session to the iPhone, you'll need to move the syslogd executable to an alternate location, kill the old instance, and start the new one with a few parameters.


# cd /usr/sbin/
# mv syslogd syslogd.mine
# launchctl stop com.apple.syslogd
....wait for about 5 seconds....
# /usr/sbin/syslogd.mine -bsd_out 1 &


Syslogd should now use the new /etc/syslog.conf file that you just created when it starts up. You can check if it's running properly:


# ps -ax |grep syslog
110 p0 S 0:02.91 /usr/sbin/syslogd.mine -bsd_out 1
#


Now fire up Splunk, and hit your instance of it in a browser: http://localhost:8000. Click on the 'admin' link in the top right, click on the 'data inputs' tab at the top, 'network ports' just below that, and then click on the 'add input' button to the right.

Click on the UDP radio button under 'source'. The port listed should change to 514. Click on the 'add' button at the bottom. You should now be getting data coming into Splunk on UDP port 514. Grab some coffee whilst Splunk eats ALL the logfiles coming in from the iPhone.

Now let's get Tinyproxy serving requests for Safari on the phone and logging through syslogd. Check that Tinyproxy is running on the iPhone first:


# ps -ax |grep tiny
354 ?? S 0:00.10 /usr/bin/tinyproxy
355 ?? S 0:00.00 /usr/bin/tinyproxy
1428 p1 S+ 0:00.01 grep tiny


Edit tiny's configuration file to set his logs to go to syslogd. Keep in mind there is more to the config file than the few lines that I'm showing.


# vi /usr/local/etc/tinyproxy/tinyproxy.conf
~
# log only errors
#Logfile "/var/log/tinyproxy.log"
#LogLevel Info
Syslog On


Now on the iPhone, go to settings..wifi networks....http proxy. Enter the host as 127.0.0.1 and the port as 8080, just as you see in the screenshot below:

Lastly, kill Tinyproxy so he'll start logging correctly. He restarts automagically, so all you need to do is kill the process ids:


# ps -ax |grep tiny
354 ?? S 0:00.11 /usr/bin/tinyproxy
355 ?? S 0:00.05 /usr/bin/tinyproxy
1651 p1 S+ 0:00.01 grep tiny
# kill -9 354 355
# ps -ax |grep tiny
1654 ?? S 0:00.01 /usr/bin/tinyproxy
1655 ?? S 0:00.00 /usr/bin/tinyproxy
1657 p1 S+ 0:00.02 grep tiny
#


That should be about it. You should have Splunk filling up with logs that contain web requests being requested by the Safari browser on your iPhone. Don't forget to restore the syslog plist file, reboot, and fix it to lock after a few minutes timeout.

• A pipeline is a thread of execution that lives within the splunkd process. Each pipeline executes a series of processors, each one which operates on data. The data is created when the first processor on the pipeline reads it from some input (like tailing a file, or receiving it on a network port). Each processor then does something to the data. Eventually, the data gets indexed and execution is returned to the first processor to get more data again.

• Pipelines are connected via queues. A queue output processor (the last processor in a pipeline) puts data on to a queue and blocks if the queue is full. A queue input processor (the first processor at the top of a pipeline) gets the data item from the bottom of the queue and sends it on down the pipeline. If there is no data, it blocks waiting for some to be put on the queue.

Enough already. Go watch the video. So, I decided that I'm tired of drawing these diagrams and wrote some code to produce them for me.

I Implemented some python code that took the composite.xml file, parsed it and produced a .dot file. Composite.xml, for those of you who don't know is an amalgamation of all pipelines and processors in the system. It represents the current (or last) runtime environment for Splunk. It lives in $SPLUNK_HOME/var/run/splunk.

I then took the resultant .dot file and ran it through graphviz. After lots of tweeking, here is what I came up with. Click on the image to see a larger version which is actually readable.

Results (click to enlarge)

Python Transformation Code

Untar this. It's only a single python file, but this blogging software wouldn't let me upload a .py file.

viz.tar

Future Work

• Annotate the graph with run time statistics like average per-processor timing, average queue size, max queue size, etc. This would require looking at the logs.
• Launching this from Splunk, firing off the python along with the metrics data pre-sifted ala Splunk.

Got more ideas? Please post them here.

My boss just told me, "Amrit, I have a camera on my computer. And when I'm at home, anytime you want, I can turn on the camera and you can watch."

There was more, but I think my ears reflexively closed in on themselves.

:/

Splunk is extending it's reach into extremely large deployments involving thousands of machines and devices across multiple data centers. The framework team is responsible for making Splunk excel in these challenging environments. If this sounds interesting and you want to work with some extremely talented people, please drop me some email.

Framework Architect / Senior Engineer

We are looking for a highly motivated engineer who will be responsible for driving the design and implementation of Splunk's network management, scalability, and distributed deployment technology. The right candidate is fluent in C++, high performance networking and concurrent / multi-threaded design.

Qualifications

• Minimum 5 years of relevant industry experience
• Expert C++ knowledge, deep understanding of design patterns and experience building clean external API's.
• Significant experience with multi-threaded design and implementation
• Has designed &amp; implemented high throughput server systems
• Practical experience with network protocols and complex topologies
• BS/MS Computer Science / Engineering
• Excellent verbal and written communication skills

So Erik did a pretty good job of describing the environment here at splunk.
The people here are great and lots of fun, there are some great problems
just begging to be solved, we need more monkeys on them typewriters

Poker games, golf, visits to the jackson arms, beer pong, foosball
(Raffy really needs a challenge )

Don't worry about that collage bit http://en.wikipedia.org/wiki/Collage

Erik insists everyone draw a picture of themselves in crayon, but really
who doesn't ask for that in a serious interview these days.

In the coming weeks I'm going to be working on a way to allow people to
plug in their own auth systems. We've had requests running the gamut from
the normal stuff like PAM, RADIUS etc to carrier pidgeon and bob's trusty
auth system. The most common thread of all these is that they are all scriptable.
You folks know your own auth systems. We'll throw this in the unstable
release/dev branch that we'll be launching and hopefully get some feed back
from you folks to fine tune it before we put it into stable.

Now that I've said that in public I'm well and truly screwed and will have to do it.

A friend of mine, Sean Dick, showed me a version of this idea using Splunk on Linux and a program called 'apci'. As I'm a Mac fanboy of sorts, I dug up a shell script for the Mac that will print out a single logfile-like line containing laptop battery information, including amp draw, amp-hours left, and more. It's aptly named 'battery', and you can download it here.

I suggest you put battery in a directory under your home directory, say something called 'scripts'. Head into 'terminal' to start the dirty work.

Here's an example output line from 'battery short':

G4:~ kord$ ./scripts/battery short
2007-10-07 18:34:27 1 _________i__ 11.232V -1.454A 2.788Ah of 4.720Ah (59.1%) of 4.400Ah (107.3%) 13 cycles

The line of underscores with an 'i' in it are the battery flags set. 'i' means my battery is installed. Duh. Other flags include whether the lid is closed, the battery is on fire, or it's just on the charger. See the battery.rtf file for more information on the flags. I have a G4 laptop, but just got my battery replaced for free! Only 13 cycles on it so far!

Splunk eats logfiles, so you'll need to get a logfile rolling on your battery output. I'm going to assume you know how to use vi (text editor) do the rest of this work.

You'll need to set up a cronjob to create the logfile and continue logging to it every so often. Switch to root and create a logfile for battery in /var/log:


G4:~ kord$ su
Password:
G4:/Users/kord root# cd /var/log
G4:/var/log root# touch battery.log
G4:/var/log root# chown kord battery.log
G4:/var/log root# ls -la battery.log
-rw-r--r-- 1 kord wheel 0 Oct 7 18:45 battery.log
G4:/var/log root# exit
G4:~ kord$


Now use 'crontab -e' and put in a line that looks something like the second line of this:


G4:~ kord$ crontab -l
* * * * * /Users/kord/scripts/battery short >> /var/log/battery.log


That will cause the battery script to run once a minute and append it to the battery.log file in the log directory. After a few minutes tail the logfile with 'tail /var/log/battery.log' and make sure you've got data in there. Also, I've edited my own crontab, but you could elect to do it as root (thus skipping the chown step above).

Obviously you will need Splunk installed to chart the battery usage out of the logfiles. If you haven't installed it already, there's a free version up on the website you can download. Follow the instructions for installing it on OSX.

Assuming that you installed Splunk in in '/Applications/splunk/' you can do the following to start it:


G4:~ root# cd /Applications/splunk
G4:/Applications/splunk root# export SPLUNK_HOME='/Applications/splunk/'
G4:/Applications/splunk root# ./bin/splunk start


Now you'll need to download my addon for Splunk, which is basically a bundle of configuration files. For reference, I also put the battery script in the tar file, along with an example crontab file. To get the bundle in the right place, start by un-taring it:


G4:~ kord$ tar xvfz battery.tar.gz
battery/
battery/addon.conf
battery/bin/
battery/bin/battery
battery/bin/battery.rtf
battery/bin/crontab.example
battery/props.conf
battery/screenshot.jpg
battery/transforms.conf


Now move it to the correct location in Splunk's directory:


G4:~ kord$ su
Password:
G4:/Users/kord root# mv battery /Applications/splunk/etc/bundles/


And restart Splunk now:


G4:/Users/kord root# /Applications/splunk/bin/splunk restart


We'll spend the rest of our time in a browser, using Splunk's kick-ass web interface.

If you left the default port alone, you should be able to fire up Firefox and hit http://localhost:8000 and see the initial login screen (or not if you are using the free version). I'll leave the particulars of getting to the initial search interface on Splunk to you.

Add the battery.log file to the list of files Splunk monitors. Click on 'admin', then click on the 'data inputs' tab. Click on the 'Add input' link to the right of 'Files &#038; Directories' at the bottom. Leave the data access to 'tail' and give the full path to the logfile - '/var/log/battery.log' in my example above. Host can be constant, DNS name doesn't matter, and set the source type pulldown to '_battery'. Remember, this sourcetype won't be in the list until you install the battery bundle.

Click on 'add' to add the source type. Go get a cup of coffee while Splunk eats this and other files on your computer and builds the index.

Back from the caffeine, you should now click on the 'splunk>' logo at the top left. Type in the following in the search bar, sans the quotes: 'source::/var/log/battery.log'. Click on the 'fields' pulldown on the left and check a few extracted fields, such as battery_ah_remaining, battery_draw, battery_percent, and battery_volts. Click on 'fields' again to close and reload with the extracted fields showing.

You should get something that looks like this:

If you have about an hour's or so data logged, try entering 'source::/var/log/battery.log | timechart avg(battery_draw)' in the search box at the top to generate a report for the last 60 minutes.

Here's what my amp draw looks like for the last 3 hours:

The move 'up' in the graph halfway through is actually a drop in amps drawn on the battery when I restarted Firefox. The cause? Firefox had a Flash game running in another tab, and it had eventually heated up the processor enough to kick on the fans!

Here's another one, showing the evidence of me having a newer battery installed - almost five hours of continuous usage after 4PM, with only a few screen sleeps:

It's interesting how the laptop charges at a rate almost the same as it discharges. It preserves battery life doing it that way, especially with the new lithium-polymer batteries.

See what else you can dig up about your battery. Try charting with some of the flags that are set - like how often the charger is on the laptop, or what the draw rate is if you have the screen clamshell closed.

Indexing Architect
We're looking for an exceptionally talented engineer to drive the design, implementation and maintenance of our core indexing and search technology. The right candidate will have significant experience writing high performance C/C++ code that interacts with the file system at low levels.

Qualifications

• Minimum 5 years of relevant industry experience.
• Expert level knowledge of C/C++ programming and a deep understanding of multi-process, highly concurrent software design.
• BS/MS Computer Science/Engineering. PhD is welcome.
• Experience in algorithm and data-structure design.
• Excellent verbal and written communication skills.

Search and Indexing Engineer (all levels)
We're looking for an exceptionally talented engineers from recent college grads to seasoned software veterans to contribute technically to Splunk's Search and Indexing team. The right candidate will care deeply about writing high-quality, efficient and maintainable software.

Qualifications

• Strong background in C/C++ programming.
• BS/MS Computer Science/Engineering or related field. PhD is welcome.
• Understanding of algorithmic complexity and data-structure tradeoffs.
• Excellent verbal and written communication skills.

Data Mining Architect
Do you love data? Terabytes of semi-structured, inconsistent, machine-generated data? If you're creative and have inspired ideas of how to summarize, group and link this data, we're looking for you.

Qualifications

• PhD in Statistics, IEOR, Computer Science or related field.
• Strong modern AI and statistics background.
• Solid command of one or more scripting languages.
• Understanding of algorithmic development and data-structure tradeoffs in C/C++.
• Strong publication history.

By the time 2.1 was on its way, we'd decided to switch to native packages, and our list of platforms had expanded to include Solaris on Intel, with several more on the horizon. We also wanted to provide the "rail tarball" distribution we continue to support, in part so that QA could get started before the packaging automation was complete.

What is that packaging automation, you might ask? Obviously writing custom code to package each platform (not to mention spec or pkgmap files in each platform's native format) was not a very maintainable solution. Instead we use a locally modified version of Easy Software's EPM package manager. After a little work, EPM lets us use a common set of list files to create relocatable packages using common pre- and post-install scripts across all of the 9 platforms we now build on. We're able to control every file and permission that goes into the packages, and in most cases we can add packaging for a new OS platform with a minimum of work (for something very different we haven't previously had in house, like AIX, more time might need to be spent cleaning up EPM's support for the platform). We've piggy-backed creation of the "rail tarball" distributions on the EPM list file structure, so those packages too are completely defined. EPM itself is built during the Splunk build process like any other 3rd party dependency, so any new patch to the tool are available to the build systems almost as soon as it's checked in.

The downside to this is a little loss of flexibility for some platforms; trivial changes to the RPM spec file or FreeBSD ports originpath have usually required modifying EPM's source. It's not a bad trade-off, though.

EPM has recently been made open source; check it out at http://www.epm-home.org . If you're interested in my patches, feel free to drop me a line, but in some cases superior patches have been contributed at epm-home.org.

Building a configuration system on the surface seems boring. If I went and showed the sales guys how cool my configuration system was they would roll their eyes back into their heads. Put some rotating, flashing thing on the GUI and they think you're the coolest, most creative developer around. The fact is that a good configuration system makes a huge difference to a product. In fact, it can make or break it in some cases.

Indulge me in allowing me to share a typical "configuration system lifecycle". Please tell me if this seems familiar to you. I have personally gone through this many times.

• Version 1.0 - simple configuration language, usually XML. Why? Because you need to get something up and running quickly. XML has tons of parsers, validators, etc. Users of this early release need to edit the configuration files using a text editor. They need to restart the system every time a change is made. The developer states that this is fine - the product is "not intended for use by people that can't use an editor". Fuck em'.
• Version 1.5 - The next release has some really complex configuration. However, it's still only modifiable via a text editor. Maybe flow control is introduced. Changing a configuration in the wrong way causes very bad and very weird things to happen. Customer Support gets lots of calls. There is no way to tell what a customer changed and what the default configuration was supposed to be without comparing the two configuration files side by side.
• Version 2.0 - We need an adminstration GUI so people can configure this without have to call support every single time! So a GUI is added. Every administered item is coded into the server and into the GUI because every configuration has different validation, different things to check, etc. The customers are much happier. Until graybeard decides he hates the GUI and insists on using emacs. The GUI and emacs don't get along very well. Things break again.
• Version 2.5 - The executives decide that we need a way for "the community" to build widgets that other people can use. They need to package these widgets up in some way that they can be downloaded and added to the system without disturbing local and default configurations. The engineers decide to use layering to separate these 3 things out. But layering in XML is nasty and people will get confused. So out with the XML to something "simpler". Boy did this open a can of worms. All the different parts of the system need to be modified to handle the new configuration syntax. We are just about ready to ship. Boy is this code base different - "Oh SHIT! We forgot we need migration scripts!". So they are frantically built and hastily tested. The product ships. Customers complain. Not only do the migration scripts hork periodically, but the configuration language is new to them.
• Version 3.0 - The server engineers are adding lots of new features to support customer requirements. Unfortunately, every new feature needs custom GUI and CLI work to handle the administration of that feature. This is simply not sustainable, so it's been decided to data drive the GUI and CLI from a specification file that describes the syntax, the interdependencies, etc for each configuration item/file. Furthermore, the community is going gangbusters, but downloading new widgets requires a restart of the server. So does all configuration changes. Once again every part of the system is changed to handle this dynamic configuration. Man is this hard - "what do I do with the data that is already in the queues when the queue is supposed to be shrunk in this re-configuration, asks one of the brightest engineers?" Hmm.

You get the idea.
So here in a nutshell is a list of reasons why configuration systems are so difficult. I'm sure you can add more:

• They are actually small languages. I have seen XML, simple linear lists of attribute/value pairs, scripting languages with flow control, strange and weird languages like in sendmail, etc.
• They need validation so they don't break the system
• If there is GUI or CLI access, they need to be dynamically updated
• Consistency between updates is critical so that someone editing a config file using via doesn't collide with someone using the GUI.
• They need to be migrated from version to version or need some kind of backward compatability
• They can be layered so local changes override system defaults
• They need to be extensible so ultimately 3rd parties can develop configurations that are add-ons
• They need solid documentation - ultimately self generating.
• They should be data driven such that every time someone invents something that needs new configuration, the GUI and/or CLI doesn't need new code.
• They need to support dynamic loading with no system restarting
• They may need to support versioning in systems that are composed of modules, each which may be independently revved.

Conclusion

Configuration systems are often overlooked, but can be the core of an entire system. There is no substitute for a really good one. It's almost impossible to get it right the first time, but you must really think long and hard about where you want it to go and what you want it to become.

Yes. I copped out. I didn't tell you how to do these things. I didn't tell you where you can look on SourceForge to find the ultimate configuration system so you don't need to re-invent the wheel yet again. That is because there is none - at least not that I know of. I have some ideas on how to build a generic configuration system that if open-sourced could save engineers months of time, but that is the topic of a different post.

More video formats are available from Splunk's Tech Talks section.

I thought it was a great idea too, what do you think?

Features are cool. They make you sound smart. Whether you're a customer talking to a sales guy, or an engineer fleshing out an idea you had. New stuff tends to show up in sentences as the word 'feature'. It's exciting. Sure it has a certain cost in speed or something. It tends to not color entirely within the lines. But that's OK. It's new, therefore it's cool.

Jumping forward many years though, everything at some point was new, and gets old and those costs start to suck. After a while these intangibles have pretty much been sacrificed away and you're in large-company hell, sitting in endless meetings trying to figure out how and when everything got all bloated and slow.

So, we're trying to swim against this current as we scale (no kidding). We're trying to prioritize speed and simplicity. We're trying to keep talking to users in the trenches. We're fighting off checkbox-itis, we're trying to have new corner-case features built in offshoot, quasi-standalone manners, we're trying to use the extensible architectures we have, and create more of them when needed.

In short, we're trying to keep that thing about Splunk that is cool. That you can get going quickly, you can set it up quickly, you can change directions quickly. It's yours to drive. When you want to do something with Splunk for the first time, it generally makes sense and doesnt take very long.

So enter YOU! We sometimes suck at all this, and could use your help to suck less. And if you the user want to talk to us, about how you use Splunk, what searches you run with it, what stuff you've found easy, what you've found hard, we want to know. It's not terribly hard to figure out my email address since you know my name. So email me. or email ui. Really. =)

We have some nifty but lightweight web-ex type stuff that just requires a browser, and we can do a 10min conference call and watch you drive your splunk instance around from the safety of our desks. Email me for an invite.

So my history at splunk - I started here in March '05. First UI Developer, inheriting the front end built by our notorious founder Erik Swan. They brought me in as a dHTML guru and gave me free reign (crossed fingers notwithstanding). But for better or for worse Splunk has always been pretty different on the client-side. Even the alphas and private betas all were all client-side XSLT and had that holy crap moment where you wonder why the hell everything is clickable and lighting up on mouseover.

Then during the sprint to 3.0 we ran off in even crazier directions, and did all the things we'd talked about doing, but held back from (eg endless pager, free form charting in Flash, rethinking the timeline interactions, replacing the tabs with more compact layers).

From this point forward though, there will be more building out and less building up if that makes sense. ie no more monolithic single all powerful UI, but rather links between quasi-standalone bits. And on the monolith instead of bolting on new features Instead we'll be solidifying things, cleaning, improving, fixing.

That said, there will still be a lot of unusual and useful interactivity. Actually probably more so overall if the monolith-maintenance burden falls as expected.

So interesting times here at Splunk. More news as it comes.

And oh, we are hiring. We are extremely hiring. If you are interested, or you know someone who's interested, or a friend of yours is really smart and you think he needs a new job... Send them our way.

External view:

Internal view:

Designing an AJAX application that meets enterprise scalability and performance requirements presents technical challenges that aren’t addressed by traditional AJAX frameworks. This session will highlight the techniques used in Splunk to address handling large amounts of data in the browser, persistent multi-panel state management, interface customization and localization, and interactive DOM-accessible graphics support. By leveraging existing, though less common, techniques such as iframe-style AJAX, in-browser XSLT, and contextual CSS, modern browsers can provide a compelling interface without the need for a thick-client installation.

Come by and say hi.

On a daily basis, I pay homage to the wonder that is Blue Bottle Coffee espresso, which flows freely - some would say excessively - from our kitchen. The benefits to productivity that this fine coffee bestows upon the dev team is enormous, easily eclipsing other contenders such as video games or foosball. Of course, there were some hurdles to get to this point, namely somebody pouring M&#038;Ms into the bean grinder of the super-automatic that was previously in service. The result was a pitiful molten mess of chocolate, beans, plastic, and gears. And, of course, the perpetrator was never discovered. So the only recourse was to beef up the machinery and move to a true commercial setup: a La Spaziale, Mazzer Mini, and freshly delivered Blue Bottle. BB even asked us what hardware we were running, and sent us the most compatible beans. Brilliant.

Process is hacking your iPhone and install ssh. Enable syslogd by the following method. (Thanks to core on #iphone)

Then edit /etc/syslog.conf and append *.* @loghost

Restart syslogd and you're set.

Then just set splunk up to listen on 514/UDP and you have iPhone logs.

Interesting bit found? launchd, the service that starts up the daemons on the iPhone just keeps respawning services. The iPhone lacks a standard service control mechanism such as the sysv-compatible init process.

The syntax is simple:

/opt/splunk/bin/splunk &lt;command&gt; [&lt;subcommand&gt;] &lt;params&gt; -uri https://my2ndSplunkBox:8089

The key here is the -uri parameter, which instructs the PCL to send all SOAP requests to the specified server. There are 3 pieces to the parameter: protocol, host, and port.

The protocol must be one of http or https, depending on whether or not SSL is enabled on the Splunkd port. Most users will want the latter, as recent versions of Splunk enable SSL on this port by default.

The second part is the hostname or IP address of the host that the remote Splunk server is running on. This should need no real explanation - in this case, the remote server has the hostname my2ndSplunkBox.

The last part of the argument is the Splunkd port (aka the management port). Note that this is not the port that's used to reach the web interface, but the port that Splunkd listens on for incoming SOAP requests. If you're unsure of what this port is, try the default, which is 8089. Alternatively, splunk show splunkd-port will display the Splunkd port that the current server is listening on.

As a practical example, one can add a tailed data input on the /var/log directory of host my2ndSplunkBox with the following command:

splunk add tail /var/log -uri https://my2ndSplunkBox:8089

The only caveat to this feature is that if you're logged into your Splunk server via splunk login, you will have to re-authenticate when sending commands to the remote server (and once again when you resume targetting your local server by leaving off -uri). Workarounds include using the -auth parameter or the SPLUNK_USERNAME and SPLUNK_PASSWORD environment variables, but these are better left to a later post.

Hi, I'm Amrit, the main CLI (Command Line Interface) and PCL (Python Control Layer) guy here at Splunk. This means that I maintain our more common bash scripts (bin/splunk &#38; friends), and our Python support scripts (site-packages/splunk/clilib/), which do the heavy lifting for a number of CLI &#38; Web UI features.

These aren't the only things I work on, but they are the parts of the Splunk codebase that have consumed most of my time since starting here in December 2005. I should also mention that Ivan Tam (no blog.. yet..?), who now works on the SplunkWeb UI, helped write the first implementation of the PCL during mid-2006.

Every now and then I'll post some tips &#38; tricks related to the things I'm working on, which you'll hopefully find useful.

KTHXBAI

5.1 Dates and Time

It has been found that some network administrators like to archive
their syslog messages over long periods of time.  It has been seen
that some original syslog messages contain a more explicit time stamp
in which a 2 character or 4 character year field immediately follows
the space terminating the TIMESTAMP.  This is not consistent with the
original intent of the order and format of the fields.  If
implementers wish to contain a more specific date and time stamp
within the transmitted message, it should be within the CONTENT
field.  Implementers may wish to utilize the ISO 8601 [7] date and
time formats if they want to include more explicit date and time
information.

Additional methods to address this desire for long-term archiving
have been proposed and some have been successfully implemented.  One
such method is that the network administrators may choose to modify
the messages stored on their collectors.  They may run a simple
script to add the year, and any other information, to each stored
record.  Alternatively, the script may replace the stored time with a
format more appropriate for the needs of the network administrators.
Another alternative has been to insert a record into the file that
contains the current year.  By association then, all other records
near that informative record should have been received in that same
year.  Neither of these however, addresses the issue of associating a
correct timezone with each record.

IMHO, this is backwards. We shouldn't require developers to put the year in the content field or have people post process logs to include the year.. Syslog should properly write out the year.

This is a change in three places, but fortunately very fast to make, and all in the same file.
$SPLUNK_HOME/share/splunk/search/dynamic/main_ui.html

Note: The example here will set your UI to search only the past 6 hours by default. After doing this it should be easy to see how to change it to search 1 day, or 45 minutes etc...

Note: Also you dont need to restart the front end to see these changes, but you DO have to refresh your browser by clicking the refresh button up top.

step 1) around line 70, change
&lt;div class="#productVersion#Version landingPageState #userType#User noTimeFields eventsTab relativeTimeMode #dynamicallySetStates#" id="outerWrapper" /&gt;

to
&lt;div class="#productVersion#Version landingPageState #userType#User eventsTab relativeTimeMode #dynamicallySetStates#" id="outerWrapper" /&gt;
(basically this removes the 'noTimeFields' state so the time controls are now open by default)

step 2) around line 122 of the same file, change
&lt;input type="text" id="relStartTime" /&gt;

to
&lt;input type="text" value="6&#8243; id="relStartTime" /&gt;

(now the UI will load with "6&#8243; already entered into the relative start field)

step 3) around line 125, still in the same file, change
&lt;option value="hours"&gt;Hours ago&lt;/option&gt;

to
&lt;option value="hours" selected="selected"&gt;Hours ago&lt;/option&gt;

(this means that hours will be selected by default. instead of minutes

That's it. You're done. Refresh your browser and the UI will now restrict it's searches to the most recent 6 hours by default. If you really only ever care about the last 2 hours, switching it to 2 hrs may speed you up even more.

Again, to reiterate Mark's qualifier - this is all assuming you understand that by doing this, you send users and passwords in clear text and the risks involved.

So, uncommenting the 2 lines as described in Mark's post will only get you the first part, ie the ability to send a GET request that logs you in. We've had people ask if that request can go further and also return results right away for a particular search they also pass in. Obvious request but somehow we didnt anticipate it.

So until we wrap this feature up in a bow in a release, once again this involves editing python by hand. And this time it's more than just uncommenting two lines. It's cut and paste, and if you know python you know that tab-indentation is meaningful, and this seemingly simple action can be deadly. You have been warned. Back up the file and proceed carefully.

Alrighty, still with us? =) Find the 2 lines that Mark blogs about uncommenting. (this will be XMLResource.py, line 395 - 400 ish depending on which 2.1 release this is)

Now replace those two lines with these lines below. NOTE: REPLACE HYPHENS WITH SPACES. wordpress seems to insist on removing leading spaces.

--------if ("usr" in request.args) and ("pwd" in request.args) :
------------logger.info("user is attempting login on GET")
------------if ("q" in request.args) :
----------------logger.info("user attempting login on GET is requesting redirection to a permalink")
----------------sessNS = request.getSession().sessionNamespaces
----------------sessNS["postLoginRedirect"] = "/?q=" + request.args["q"][0]
 -  -  -  - return self.render_POST(request)

now restart the python front end using splunk restartss (a full splunk restart is not necessary)
And now you'll have the ability to embed URL's like this in the webapp of your choice

http://your.host/login?usr=username&#38;pwd=password&#38;q=interestingTerm1%20interestingTerm2

UPDATE - -

as pointed out in the first comment (thanks!!) the above snippet will happily fall into a recursive loop if the auth information it's given is incorrect. New improved version below: (AGAIN, REPLACE LEADING HYPHENS WITH SPACES)

--------if ("usr" in request.args) and ("pwd" in request.args) :
------------logger.info("user is attempting login on GET")
------------sessNS = request.getSession().sessionNamespaces
------------if ("cannotConnectToSplunkd" not in sessNS and "error" not in sessNS) :
----------------if ("q" in request.args) :
--------------------logger.info("user attempting login on GET is requesting redirection to a permalink")
--------------------sessNS["postLoginRedirect"] = "/?q=" + request.args["q"][0]
 -  -  -  -  - -return self.render_POST(request)

So. the link below is your friend. (If you've used bookmarklets before you know what to do. Otherwise, read on. )

Instead of clicking this link though, right-click it or option click it, and choose 'bookmark this link'.
splunk 30 second refresh

Once you've done that, then whenever you have Splunk loaded, clicking that bookmark will run the tiny little script, and the upshot is that the UI will start autorefreshing in 30 seconds and every 30 seconds thereafter.

And if you want to change the 30 seconds, edit the bookmark, find the 30000 and change it to whatever.

Thanks to my acquisition-happy former employer, Symantec, I've seen a variety of startup approaches to release engineering. Most frequently it seems some senior developer has a bug up you-know-where about how the build system should work, and some poor junior developer or sysadmin type person dutifully does the drudge work (usually by hand). At other sites, some very diligent and detail-oriented person creates and executes a process with a great deal of record-keeping and attention to detail but often not a lot of automation. Consistency across different build platforms usually isn't a strong point.

Here at Splunk, things are a bit different. I called myself the plumber in the title of this post because that's how I see my job: I create and maintain the plumbing that produces consistent, reproducible Splunk builds across all of our platforms, with as much visibility as I can muster. I see my contribution more as enforcing process through tools - ideally, tools that enable process in a way that is more convenient for everyone than "doing it wrong" - rather than personally pushing all the buttons and scribbling in all the logbooks. And I've had the good fortune to come into a culture that encourages this approach.

Whew. That's a mouthful for an introduction. In the near future I hope to write a bit more about how the plumbing works, and some neat tools I've found along the way. I'm sure y'all will be waiting with baited breath.

To configure selinux to allow splunk to run, you need to run the chcon command on the splunk lib directory. Here is what you type :

chcon -c -v -R -u system_u -r object_r -t lib_t $SPLUNK_HOME/lib 2&gt;&#38;1 &gt; /dev/null
You can also disable the check when splunk starts by adding this line to the $SPLUNK_HOME/bin/setSplunkEnv script

export SPLUNK_IGNORE_SELINUX=1

We've had a few people ask for this. Its going to be in the documentation eventually, but until then here is how you do it.

Edit $SPLUNK_HOME/etc/myinstall/search.xml

Change :

&lt;updateCheckerBaseURL&gt;http://quickdraw.splunk.com/js/&lt;/updateCheckerBaseURL&gt; &lt;updateCheckerBaseURL&gt;0&lt;/updateCheckerBaseURL&gt;

(2.1)

$SPLUNK_HOME/share/splunk/search/static/js/update_checker_pro.js.

At the top of the file, and within that same setup function, comment out these two lines:
createUpdateCheckerScriptlet();
setTimeout('possiblyFallBackToCannotConnectMessage()', 5000);

Assuming you understand that by doing this, you send users and passwords in clear text and the risks involved.

There is a way to do this through http GET, but it requires modifying a bit of python.

Edit line 395 of XMLResourse.py located in $SPLUNK_HOME/lib/python2.4/site-packages/splunk/search/XMLResource.py

def render_GET(self, request) :
# backdoor so scripts can auto-login just with a GET request instead of having to craft a proper HTTP POST. Doesnt help said script keep track of the cookie, which is the hard part.
#if ("usr" in request.args) and ("pwd" in request.args) :
# return self.render_POST(request)
logger.debug("LoginResource.render_GET")
sessNS = request.getSession().sessionNamespaces

Uncomment out the if and return lines and restart splunk.

To log in, you would enter this URL

http://your.host/login?usr=username&#38;pwd=password

Note that this will negatively impact indexing performance but it should work until we get this behavior baked into splunk.

First up I've created a python script that calls socket.gethostbyaddr to resolve the hosts. It will also cache the results so that the performance hit for dns misses is reduced.
So copy and paste the following into your favorite editor and save it to &lt;SPLUNK_HOME&gt;lib/python2.4/site-packages/splunk/pyHostNameResolve.py . This directory is where the dynamic loaded python will look for scripts; the filename will be referenced later in a config change.

#Copyright (C) 2006 Splunk Inc. All Rights Reserved. This work contains trade
#secrets and confidential material of Splunk Inc., and its use or disclosure in
#whole or in part without the express written permission of Splunk Inc. is prohibited.

from pipeline_data import PipelineDataWrapper #This is a virtual module/class that gets inserted into the python namespace at runtime by splunk
import traceback
import socket

#Set global variables
HOST_KEY = "MetaData:Host"

HOST_RESOLVE_MAP = {} #cache so we don't have to call gethostbyaddr ( expensive ) every event

def resolveHost( pdata, confDictString ):
    global HOST_RESOLVE_MAP
    try:

        host = pdata.get(HOST_KEY)

        resolvedHostName = None

        if host.startswith("host::") :
            host = host[6:]

        if host in HOST_RESOLVE_MAP:
            resolvedHostName = HOST_RESOLVE_MAP[ host ]

        if not resolvedHostName:
            try:
                resolved = socket.gethostbyaddr(host)
                resolvedHostName = resolved[0]
                HOST_RESOLVE_MAP[ host ] = resolvedHostName
            except:
                HOST_RESOLVE_MAP[ host ] = host
                print "Could not resolve " + host
                return 1

        if resolvedHostName :
            pdata.put( HOST_KEY, "host::"+resolvedHostName )

        return 1    

    except:
        print "EXCEPTION !!"
        traceback.print_exc()
        return -1

Ok now open your &lt;SPLUNK_HOME&gt;/etc/myinstall/splunkd.xml and insert the following chunk of xml between the diskusageprocessor and the bytequotaprocessor in the indexerpipe pipeline :

               <processor name="hostnameresolver" plugin="pythonprocessor">
                                 <config>
                                         <scriptFilename>splunk.pyHostNameResolve</scriptFilename>
                                         <command>resolveHost</command>
                                         <pyContext>resolveContext</pyContext>
                                         <pyConfig><![CDATA[]]></pyConfig>
                                 </config>
                         </processor>

Ok now fire up splunk and you should start seeing your hosts getting resolved. Note that this will negatively impact performance but it should work until we get this behavior baked into splunk.
Cheers,
Brian

The answer is - omg it works well. Fast, seemingly stable, it can be pushed. Even in a big javascript front end like ours, the event handlers on svg elements pass right up into our existing framework. Some small tweaks had to be made to accomodate it, but no showstoppers. And it is rare for such a complicated thing to present so few obstacles in practice.
So thanks to Mozilla for being generally awesome, and particularly for turning on SVG in their release builds . Of course i have absolutely no idea if any svg will ever appear in the product ... We do after all have a great deal of other more mundane improvements in the works. =)

Also, my apologies for not talking about skins. I had wanted to post a big treatise on skins, but since I'm rewriting and reworking all the css right now, it would be way too cruel; any skins you made would die with the upgrade to 1.3. So Im saving the post for another day.

Until then, for the indomitably curious, suffice it to say that the 'invert' link hidden in the footer actually cycles through the skin list, and the fact that there are currently only two skins shipping in the product does not prevent you the user from hacking the front end and making a third, fourth skin, etc...

if you're feeling adventurous, put another skin file in [opt/splunk]/share/splunk/search/static/css/skins/, crack open photoshop and use all the existing skin graphics as a base to make your new skin from,
and as for how to hook up this new skin, for later 1.2 dot releases, look for the skinFileList in [opt/splunk]lib/python2.4/site-packages/splunk/search/SearchService.py

(for I think 1.2 and older builds, the list of skins was just the link tags in share/splunk/search/dynamic/main_ui.html

Im the front-end guy. From the xsl,js,html and css on the client side, up to the python in splunkSearch, I am responsible (read: to blame) for the current implementation, and also for much of the interaction design. I've worked here just over a year now, and I have no noticeable scars or weird tics to show for it, so I guess I've got that going for me.

What possessed me to come work here: Essentially all of my experience before splunk was at services companies, and for the prior 3 or 4 years specifically I had become this sort of high-throughput template builder and dhtml-specialist. Boring stuff I know, but I mention it because I came to splunk partly to get away from this. You can build lots of really complex front ends while at services companies. You can do build for flexibility and simplicity all you want, but when the project is over you never really see the code again (or worse the code never gets updated or changed by anyone and so it never evolves at all). So outside of maybe some escalated issues, you never really know what were the good parts of the implementation and what were the bad. And the development pain induced by changing requirements, the codebase evolution and accidental devolution, the day to day suffering really, you get spared from all that and that sucks.

So here at splunk I get all that too. I'm not just some head-in-the-clouds master-template builder, I'm actually the poor slob responsible for maintaining it too. =)

One nice silver lining is that sometimes I'll get rewarded with these little gems of code that have been going on a journey towards the nonsensical. Where say, at one point it did something simple and made sense, then something complicated was factored in alongside, then at 2AM before such and such release that changed, then later there was a quick tweak to address someones feedback, then another change etc... and then suddenly you look at it one day and its just a stunningly silly little thing and you throw it away and rewrite it in 5 minutes.

I'll try and post again in a couple days. There's a lot of dry topics that I would love to ramble on about, but this post is pretty parched actually, now that I read it.

So maybe instead I'll post on how to create your own skin or something.

or even better, I'll update you on recent injuries imparted by our skateboard-trapeze-of-death... (so far Rory and Marc have had the only mishaps, Rory had nothing broken, and Marc escaped with no injuries at all)


export SPLUNK_HOME=<WHERE_YOU_INSTALLED_SPLUNK>
export PATH=$SPLUNK_HOME/bin:$PATH
export LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH


Ok so now you should be good to go so fire up python. Your python version should be 2.4.2. If it's not do a "which python" from the command prompt to make sure you are using the python that shipped with splunk.
We need to do some setup before any searches can be run :


Python 2.4.2 (#1, Mar 11 2009, 21:45:07)
[GCC 4.0.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.


>>> import splunk.search.splunkTest #initialize the python internals without using twistd
>>> import splunk.search.SearchCore as SearchCore #This is the module we are going to use to issue searches

If you want to run against a remote splunk server or on different ports you can run the following :


>>> SearchCore.SearchService.gSearchService._searchEngineURL = "http://<remote_host>:<searchengine_port>"


The method on the SearchCore module that executes the queries is called runQuery and it takes two arguments.


def runQuery(queryString, userStr )


The userStr can be any string for now; in future releases it will probably be an auth token. It is the user that your searches will appear under in the searchhistory domain.
The queryString is where the magic happens .
Basically a query string contains three major elements.

QUERY : Terms following this are as you would see in the splunk web ui search box. This pulls the resulting ids into an id space internally in the query.
GET : Terms following this instruct splunk on what extract from ids in the id space into results the result space.
OUTPUT : How to format the results from the result space to output.

For a more detailed reference on the query syntax check out : http://www.splunk.com/index.php/docs?doc=developer.html&#38;vers=#58

Now for our first search :

The meta::all key is a splunk key that every event in the system will have.


>>> SearchCore.runQuery("QUERY meta::all","brian")


You will get the result "&lt;queryResult&gt;&lt;/queryResult&gt;" from this as we have not specified an OUTPUT element. Note that unless you specify a domain to run these queries in they will run in the default index ( main ).

Run :

>>> SearchCore.runQuery("QUERY meta::all OUTPUT splunkui::1.0″,"brian") # We use the splunkui output here because we want to do things that the ui does like get events ...


Now the result is :

<queryResult><eventIndexedCount>58728</eventIndexedCount>
<ids>
</ids>
<projectedResultCount>1001</projectedResultCount>
<clampedStartTime>1049204073</clampedStartTime>
<clampedEndTime>1142300808</clampedEndTime>
</queryResult>


Of course your numbers will be different.
The projected result count element is legacy and can be safely ignored.
The eventIndexedCount is the total number of events in this domain.
The clampedStartTime/clampedEndTime constrain the timerange in which results for this query may appear.

Note there is still no event output ... lets fix that :


>>> SearchCore.runQuery("QUERY meta::all GET events::0-2 OUTPUT splunkui::1.0 format::raw", "brian") #The format::raw tells the outputter to ignore all segment information


Results :


<queryResult><eventIndexedCount>19704</eventIndexedCount>
<ids>
</ids>
<projectedResultCount>1001</projectedResultCount>
<clampedStartTime>1041618608</clampedStartTime>
<clampedEndTime>1142302043</clampedEndTime>
<results type="events">
<result cd="0:1532081″>
<segtext xml:space="preserve">Oct 14 16:29:38 liftoff sendmail[20336]: i9ENTcHf020336: from=<erik@transaction-engines.com>, size=667, class=0, nrcpts=1, msgid=<416F0BE2.3060306@transaction-engines.com>, proto=ESMTP, daemon=MTA, relay=h-68-167-140-171.snvacaid.covad.net [68.167.140.171]</segtext>
<timestamp>1141691378</timestamp>
<source cd="1″ string="/opt/splunk/var/spool/splunk/maillog">
<dir>/opt/splunk/var/spool/splunk/</dir>
<file>maillog</file>
</source>
<host cd="1″>localhost</host>
<sourcetype cd="1″ base="sendmail_syslog">sendmail_syslog</sourcetype>
<type cd="38″ wob=" v:de22 t:97 t:49 t:17882122 t:2336388840 t:63489930 t:4036439400 t:0 ">
<tags><tag>transaction</tag><tag>class</tag><tag>sendmail</tag><tag>com</tag><tag>size</tag><tag>net</tag></tags>
</type>
</result>
<result cd="0:2223455″>
<segtext xml:space="preserve">Oct 18 15:14:27 liftoff sendmail[2527]: i9IMERup002527: from=<erik@transaction-engines.com>, size=3690, class=0, nrcpts=1, msgid=<41744043.3060306@transaction-engines.com>, proto=ESMTP, daemon=MTA, relay=h-68-167-140-171.snvacaid.covad.net [68.167.140.171]</segtext>
<timestamp>1141686867</timestamp>
<source cd="1″ string="/opt/splunk/var/spool/splunk/maillog">
<dir>/opt/splunk/var/spool/splunk/</dir>
<file>maillog</file>
</source>
<host cd="1″>localhost</host>
<sourcetype cd="1″ base="sendmail_syslog">sendmail_syslog</sourcetype>
<type cd="38″ wob=" v:de22 t:97 t:49 t:17882122 t:2336388840 t:63489930 t:4036439400 t:0 ">
<tags><tag>transaction</tag><tag>class</tag><tag>sendmail</tag><tag>com</tag><tag>size</tag><tag>net</tag></tags>
</type>
</result>
<result cd="0:3155870″>
<segtext xml:space="preserve">Oct 21 14:03:53 liftoff sendmail[11725]: i9LL3quJ011725: from=<erik@transaction-engines.com>, size=2663, class=0, nrcpts=1, msgid=<41782438.7060303@transaction-engines.com>, proto=ESMTP, daemon=MTA, relay=h-68-167-140-171.snvacaid.covad.net [68.167.140.171]</segtext>
<timestamp>1141423433</timestamp>
<source cd="1″ string="/opt/splunk/var/spool/splunk/maillog">
<dir>/opt/splunk/var/spool/splunk/</dir>
<file>maillog</file>
</source>
<host cd="1″>localhost</host>
<sourcetype cd="1″ base="sendmail_syslog">sendmail_syslog</sourcetype>
<type cd="38″ wob=" v:de22 t:97 t:49 t:17882122 t:2336388840 t:63489930 t:4036439400 t:0 ">
<tags><tag>transaction</tag><tag>class</tag><tag>sendmail</tag><tag>com</tag><tag>size</tag><tag>net</tag></tags>
</type>
</result>
</results>
</queryResult>


Now you can see the actual event text in the segtext element in the results.
If you want to get counts like you see in the tab headings in the splunkui you can use OUTPUT term scheduler::1.0.
This will give you the following output :


<queryResult>
<schedResults>
<eventCount>10000+</eventCount>
<hostCount>1+</hostCount>
<sourceCount>1+</sourceCount>
<typeCount>239+</typeCount>
<sourceTypeCount>1+</sourceTypeCount>
<eventtagCount>62+</eventtagCount>
<starttime>12/31/1969:16:00:00</starttime>
<endtime>03/13/2006:18:50:48</endtime>
</schedResults>
</queryResult>


Note the + marks that are the equivalent of the &gt; signs in the ui that tell you that there may be more than what is displayed.
You may mix the splunkui and scheduler outputs in a single querystring.

Tune in next time where I'll explain some of the more advanced elements of the search language.

Brian

Fear not as there is a hidden undocumented call that you can make ! If you run the query "++cmd++::optimize" you will cause a database optimization. This call may take a while to return so use with care. Soon we will have a release with an auto-optimizer but if it's hampering your splunking right now you can create a live splunk to run every 10-30 mins that runs "++cmd++::optimize".

Laters,

Brian

So this is the start of my splunk blog.

First up I'm splunk employee #1. Way back in Sept. 2004 I joined Erik, Rob and Michael when they were still based down in the VC offices in Palo Alto. I'm responsible for searches and indexing so if you have splunks that are taking WAAAY too long to complete I'm the person that's probably responsible.

I'll post more later on what I'm coding, struggling against or just hacking on.

Brian out.

What is Splunk?

Splunk is a search server that indexes all your log files.

If you need to search and troubleshoot log files, you need Splunk. It
handles any log format, including syslog, Apache, Jboss, mysql,
oracle, router data, etc. It parses and indexes in real time.

Grep works fine. Why do I need Splunk?

grep is totally fine for small, simple, local files, but grep doesn't
work on 20GB of log files, across a dozen servers; doesn't group
multiline log messages together; doesn't unify timestamps across
files; doesn't automatically find related log events; doesn't show
histograms of log events; doesn't search gigabytes in seconds; doesn't
have a cool ajax web interface similar to google.

What are multiline log messages?

As an example, java exceptions look like this:
[source:java]java.lang.reflect.UndeclaredThrowableException
at $Proxy231.getAllAttributes(Unknown Source)
at com.collation.proxy.clientproxy.common.Module.getModelObject(Module.java:326)
at com.collation.proxy.clientproxy.server.action.ChangeHistoryModule.getDependencies(ChangeHistoryModule.java:402)
at com.collation.proxy.clientproxy.server.action.ChangeHistoryModule.getIdsWithDependencies(ChangeHistoryModule.java:386)
...
[/source]

You can't use
grep to search for java proxy exceptions because
"Exception" and "proxy" don't occur on the same line! The same
would apply to sql, router data, email, or any other multiline event.
Splunk automatically groups
multiline events into single events, so the above exception
would become one event. Splunk does this with advanced heuristics and
machine learning algorithms, as well as customizeable groupping rules.

What about unifying timestamps?

Most log files have timestamps embedded in them. Splunk understands
dozens and dozens of timestamp formats, unifying them across
timezones. Some log files write events out as GMT (Greenwich Mean
Time) some as local time such as PST (Pacific Standard Time). Some
logs can come from servers on the east coast, some from the west
coast, or beyond. By
normalizing all these timeszones in dozens of timestamp formats,
Splunk allows you to say "What happened at 11:57pm", world-wide,
across all my log files, across all my servers. "I got an error
at 1:15am yesterday. Show me the log events from all my logs just
before 1:15&#8243;.

OK, one more. What are related log events?

Suppose you see suspecious activity or an error. Just ask Splunk to
find logs related to that activity. It'll find logs that have the
same IP, UserID, URL, codes, etc. If there was a problem with an IP,
Splunk will show you all the related events for that IP; same for
UserID, URL, or any other code. You can even ask Splunk to show you events sorted
by how unexpected they are!

How much does Splunk cost?

The Splunk Personal Server is Free. Give it a try.

How can I get Splunk?

Go to: www.splunk.com