CodeBits - Tested Complex Code!

If you use AccumuloRowInputFormat but your job is hung and no scans are running .... you might have forgotten to use values.next().

2013-05-24T10:13:00.000-04:00

Infinite loops. Haven't seen one in two years.

Pointer to "Jump Point Search Explained" Page

2013-05-23T07:17:00.002-04:00

http://zerowidth.com/2013/05/05/jump-point-search-explained.html - This explanation is very well done.

Understanding Markov Networks

2013-05-14T09:24:00.000-04:00

http://izbicki.me/blog/markov-networks-monoids-and-futurama - I just found a very readable introduction to Markov networks. Don't be scared by the Haskell code, the concepts are all well-explained.

Adding a tee command to Accumulo Shell

2013-05-12T11:46:00.000-04:00

An afternoon project. There are hacks involved. Do not use in production!

Some of the iterators that I've been writing are designed to create a problem-oriented dataset; a limited view into the larger dataset. Once the iterators are put into place in the shell, there isn't a way to easily materialize that sub-set of the data. I'm not even sure it makes sense to materialize it, but it was interesting to experiment with the code.

Because this project was so specific to my whim, I don't feel it's right to add to the official code base.

My first step was an update to the Shell.java file. I added "new TeeCommand()" to the external[] Command array. Then I added a private String attribute called 'teeTableName". The last change was to the printRecords method. This change was a hack.

Formatter formatter = FormatterFactory.getFormatter(formatterClass, scanner, printTimestamps);
if (formatter instanceof TeeFormatter) {
    ((TeeFormatter)formatter).setConnector(connector);
    ((TeeFormatter)formatter).setTeeTableName(teeTableName);
}

The TeeCommand class is fairly simple. The only interesting part of the execute() method. You'll note that the teeTable can't be the same as the current table in the shell. And it is automatically created if it does not exist. Another point to note is that the formatter for the current table is changed *globally*. Another hack. And a dangerous one. I don't see a cleaner way to assign the formatter with larger changes to the Shell class.

@Override
    public int execute(String fullCommand, CommandLine cl, Shell shellState) throws AccumuloException, AccumuloSecurityException, TableNotFoundException, TableExistsException {
        String tableName = cl.getArgs()[0];
        String currentTableName = shellState.getTableName();
        if (currentTableName.equals(tableName)) {
            throw new RuntimeException("You can't tee to the current table.");
        }
        if (!shellState.getConnector().tableOperations().exists(tableName)) {
            shellState.getConnector().tableOperations().create(tableName);
        }

        String subcommand = cl.getArgs()[1];
        if ("on".equals(subcommand)) {
            shellState.setTeeTableName(tableName);
            shellState.getConnector().tableOperations().setProperty(shellState.getTableName(), Property.TABLE_FORMATTER_CLASS.toString(), TeeFormatter.class.getName());

        } else if ("off".equals(subcommand)) {
            shellState.setTeeTableName(null);
            shellState.getConnector().tableOperations().removeProperty(shellState.getTableName(), Property.TABLE_FORMATTER_CLASS.toString());
        }
       
        return 0;
    }

The last change was to develop the TeeFormatter class. It's a copy of the DefaultFormatter except for the addition of a copyEntry method which inefficient in the extreme because it opens a BatchWriter for *every* row in the scan. I'll leave it to the reader to develop a more efficient approach. Note that I choose random number for the createBatchWriter call. More hackery!

private void copyEntry(Entry entry) {
      BatchWriter wr = null;
      try {
          wr = connector.createBatchWriter(teeTableName, 10000000, 10000, 5);
          Key key = entry.getKey();
          Value value = entry.getValue();
          Mutation m = new Mutation(key.getRow());
          m.put(key.getColumnFamily(), key.getColumnQualifier(), new ColumnVisibility(key.getColumnVisibility().toString()), key.getTimestamp(), value);
          wr.addMutation(m);
      } catch (TableNotFoundException e) {
          throw new RuntimeException("Unable to find table " + teeTableName, e);
      } catch (MutationsRejectedException e) {
          throw new RuntimeException("Mutation rejected while copying entry to tee table.", e);
      } finally {
          if (wr != null) {
              try {
                  wr.close();
              } catch (MutationsRejectedException e) {
          throw new RuntimeException("Mutation rejected while closng writer to tee table.", e);
              }
          }
      }
  }

I did not include the whole solution in this email because of length, hackery, and criminal inefficiency. However, if you want this tee command the clues above should let you write your own.

In order to help Accumulo hacking, I've updated my https://github.com/medined/accumulo-at-home project to include a 1.4.3/update-accumulo-1.4.3.sh script.

OS/X - tap touchpad with two fingers for a right click.

2013-04-03T18:32:00.001-04:00

My life is suddenly a lot easier... What else don't I know about using OS/X?

Example using Accumulo's RegExFilter class

2013-03-31T14:35:00.001-04:00

package com.affy;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.SortedMap;
import java.util.TreeMap;
import org.apache.accumulo.core.client.IteratorSetting;
import org.apache.accumulo.core.conf.AccumuloConfiguration;
import org.apache.accumulo.core.data.ByteSequence;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.iterators.IteratorEnvironment;
import org.apache.accumulo.core.iterators.IteratorUtil;
import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
import org.apache.accumulo.core.iterators.SortedMapIterator;
import org.apache.accumulo.core.iterators.system.MapFileIterator;
import org.apache.accumulo.core.iterators.user.RegExFilter;
import org.apache.accumulo.core.util.CachedConfiguration;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.log4j.Logger;

public class AccumuloRegExIteratorPlayground {

    private final Logger log = Logger.getLogger(AccumuloRegExIteratorPlayground.class);
    private static final Collection<ByteSequence> EMPTY_COL_FAMS = new ArrayList<ByteSequence>();

    public void process() throws IOException {
        final String regularExpression = "/.*";

        final SortedMap<Key, Value> input = new TreeMap<Key, Value>();
        input.put(new Key("1111", "2222", "3333", 0), new Value("4444".getBytes()));
        input.put(new Key("/1111", "2222", "3333", 0), new Value("4444".getBytes()));

        final RegExFilter rei = new RegExFilter();
        IteratorSetting is = new IteratorSetting(1, RegExFilter.class);
        RegExFilter.setRegexs(is, regularExpression, null, null, null, false);

        if (!rei.validateOptions(is.getOptions())) {
            throw new RuntimeException("invalid options.");
        }

        rei.init(new SortedMapIterator(input), is.getOptions(), new IteratorEnvironment() {
            @Override
            public SortedKeyValueIterator<Key, Value> reserveMapFileReader(String mapFileName) throws IOException {
                Configuration conf = CachedConfiguration.getInstance();
                FileSystem fs = FileSystem.get(conf);
                return new MapFileIterator(AccumuloConfiguration.getDefaultConfiguration(), fs, mapFileName, conf);
            }

            @Override
            public AccumuloConfiguration getConfig() {
                return AccumuloConfiguration.getDefaultConfiguration();
            }

            @Override
            public IteratorUtil.IteratorScope getIteratorScope() {
                throw new UnsupportedOperationException("Not supported yet.");
            }

            @Override
            public boolean isFullMajorCompaction() {
                throw new UnsupportedOperationException("Not supported yet.");
            }

            @Override
            public void registerSideChannel(SortedKeyValueIterator<Key, Value> iter) {
                throw new UnsupportedOperationException("Not supported yet.");
            }
        });
        rei.seek(new Range(), EMPTY_COL_FAMS, false);

        while (rei.hasTop()) {
            final Key key = rei.getTopKey();
            final Value value = rei.getTopValue();
            log.info(key + " --> " + value);
            rei.next();
        }
    }

    public static void main(final String[] args) throws IOException {
        AccumuloRegExIteratorPlayground driver = new AccumuloRegExIteratorPlayground();
        driver.process();
    }
}

Testing Your GrepIterator Without Running Accumulo

2013-03-30T18:58:00.000-04:00

An example program...

package com.affy;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Map;
import java.util.SortedMap;
import java.util.TreeMap;
import org.apache.accumulo.core.client.IteratorSetting;
import org.apache.accumulo.core.client.impl.MasterClient;
import org.apache.accumulo.core.data.ByteSequence;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.iterators.SortedMapIterator;
import org.apache.accumulo.core.iterators.user.GrepIterator;
import org.apache.log4j.Logger;

public class AccumuloGrepIteratorPlayground {

    private static final Logger log = Logger.getLogger(AccumuloGrepIteratorPlayground.class);
    private static final Collection EMPTY_COL_FAMS = new ArrayList();

    public static void main(final String[] args) throws IOException {
        final String term = "/1";

        final SortedMap<Key, Value> input = new TreeMap<Key, Value>();
        input.put(new Key("1111", "2222", "3333", 0), new Value("4444".getBytes()));
        input.put(new Key("/1111", "2222", "3333", 0), new Value("4444".getBytes()));

        final GrepIterator grepIterator = new GrepIterator();
        final IteratorSetting iteratorSetting = new IteratorSetting(1, GrepIterator.class);
        GrepIterator.setTerm(iteratorSetting, term);
        grepIterator.init(new SortedMapIterator(input), iteratorSetting.getOptions(), null);
        grepIterator.seek(new Range(), EMPTY_COL_FAMS, false);

        while (grepIterator.hasTop()) {
            final Key key = grepIterator.getTopKey();
            final Value value = grepIterator.getTopValue();
            log.info(key + " --> " + value);
            grepIterator.next();
        }

    }
}

The pom.xml looks like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.mycompany</groupId>
    <artifactId>AccumuloGrepIterator</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>AccumuloGrepIterator</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.accumulo</groupId>
            <artifactId>accumulo-core</artifactId>
            <version>1.4.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>0.23.6</version>
        </dependency>        
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

Running Accumulo Unit Tests On Windows

2013-03-27T19:22:00.000-04:00

Accumulo is not designed to run under Windows. Nor are its unit tests. However, if you persist in this foolishness, add the following elements to the indicated pom.xml files.

core/pom.xml

    <profile>
        <id>skipped_tests_on_windows</id>
        <activation>
            <activeByDefault>false</activeByDefault>
            <os>
                <family>Windows</family>
            </os>
            </activation>
        <build>
        <plugins>
            <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <excludes>
                        <exclude>**/AccumuloFileOutputFormatTest.java</exclude>
                        <exclude>**/AccumuloFileInputFormatTest.java</exclude>
                        <exclude>**/AccumuloOutputFormatTest.java</exclude>
                        <exclude>**/AccumuloRowInputFormatTest.java</exclude>
                        <exclude>**/AccumuloInputFormatTest.java</exclude>
                        <exclude>**/MockTableOperationsTest.java</exclude>
                        <exclude>**/AggregatingIteratorTest.java</exclude>
                        <exclude>**/IterUtilTest.java</exclude>
                        <exclude>**/CombinerTest.java</exclude>
                        <exclude>**/FormatterCommandTest.java</exclude>
                        </excludes>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>

examples/simple/pom.xml

    <profile>
        <id>skipped_tests_on_windows</id>
        <activation>
            <activeByDefault>false</activeByDefault>
            <os>
                <family>Windows</family>
            </os>
            </activation>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <excludes>
                            <exclude>**/ChunkInputFormatTest.java</exclude>
                            </excludes>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>

proxy/pom.xml

    <profile>
        <id>skipped_tests_on_windows</id>
        <activation>
            <activeByDefault>false</activeByDefault>
            <os>
                <family>Windows</family>
            </os>
            </activation>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <excludes>
                            <exclude>**/SimpleTest.java</exclude>
                            <exclude>**/TestProxyInstanceOperations.java</exclude>
                        </excludes>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
  </profiles>

server/pom.xml

    <profile>
        <id>skipped_tests_on_windows</id>
        <activation>
            <activeByDefault>false</activeByDefault>
            <os>
                <family>Windows</family>
            </os>
            </activation>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <excludes>
                            <exclude>**/TableLoadBalancerTest.java</exclude>
                            </excludes>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>

start/pom.xml

    <profile>
        <id>skipped_tests_on_windows</id>
        <activation>
            <activeByDefault>false</activeByDefault>
            <os>
                <family>Windows</family>
            </os>
            </activation>
        <build>
        <plugins>
            <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <excludes>
                        <exclude>**/AccumuloVFSClassLoaderTest.java</exclude>
                        <exclude>**/ReadOnlyHdfsFileProviderTest.java</exclude>
                        <exclude>**/VfsClassLoaderTest.java</exclude>
                        </excludes>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>

test/pom.xml

    <profile>
        <id>skipped_tests_on_windows</id>
        <activation>
            <activeByDefault>false</activeByDefault>
            <os>
                <family>Windows</family>
            </os>
            </activation>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <excludes>
                            <exclude>**/MiniAccumuloClusterTest.java</exclude>
                            <exclude>**/ShellServerTest.java</exclude>
                        </excludes>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>

How To Stop Accumulo Debug Messages From Being Logged

2012-12-29T16:01:00.001-05:00

Comment out references to A2 in conf/generic_logger.xml and conf/monitor_logger.xml. Then stop all Accumulo processes. Delete logs/*. Restart Accumulo. Done.

SOLVED: add-apt-repository command not found.

2012-12-18T23:57:00.001-05:00

Run the following to install the package with the add-apt-repository command.

apt-get -y install software-properties-common

Exampe of Storing User and Friends Information in Accumulo

2012-11-13T00:15:00.000-05:00

This example shows one way to store user and friend information in one Accumulo table. Each friend category becomes a Column Family and each friend's user id becomes a Column Qualifier. After the inserts a small query is run to loop over old friends.

package com.codebits.accumulo;

import java.io.IOException;
import java.util.Iterator;
import java.util.Map;

import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.Connector;
import org.apache.accumulo.core.client.Instance;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.client.TableExistsException;
import org.apache.accumulo.core.client.TableNotFoundException;
import org.apache.accumulo.core.client.mock.MockInstance;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Mutation;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.security.Authorizations;
import org.apache.hadoop.io.Text;

public class UserAndFriendsDriver {

  public static void main(String[] args) throws IOException, AccumuloException, AccumuloSecurityException, TableExistsException, TableNotFoundException {
    Instance mock = new MockInstance("development");
    Connector connector = mock.getConnector("root", "password".getBytes());
    connector.tableOperations().create("TABLEA");

    BatchWriter wr = connector.createBatchWriter("TABLEA", 10000000, 10000, 5);
    Mutation m = new Mutation(new Text("john"));
    m.put("info:name", "", "john henry");
    m.put("info:gender", "", "male");
    m.put("friend:old", "mark", "");
    wr.addMutation(m);
    m = new Mutation(new Text("mary"));
    m.put("info:name", "", "mark wiggins");
    m.put("info:gender", "", "female");
    m.put("friend:new", "mark", "");
    m.put("friend:old", "lucas", "");
    m.put("friend:old", "aaron", "");
    wr.addMutation(m);
    wr.close();

    Scanner scanner = connector.createScanner("TABLEA", new Authorizations());
    scanner.setRange(new Range("a", "z"));
    scanner.fetchColumnFamily(new Text("friend:old"));
    Iterator<Map.Entry<Key, Value>> iterator = scanner.iterator();
    while (iterator.hasNext()) {
      Map.Entry<Key, Value> entry = iterator.next();
      Key key = entry.getKey();
      System.out.println("Old Friends: " + key.getRow() + " -> " + key.getColumnQualifier());
    }
  }
}

How Can I Use Reverse Sort On Integer (or Long) Accumulo Keys?

2012-11-12T17:04:00.001-05:00

This entry will be very similar to my last. Here is code with debugging (i.e., the original key is retained) so that you can see the reverse sort is working.

package com.codebits.accumulo;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Iterator;
import java.util.Map;

import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.Connector;
import org.apache.accumulo.core.client.Instance;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.client.TableExistsException;
import org.apache.accumulo.core.client.TableNotFoundException;
import org.apache.accumulo.core.client.mock.MockInstance;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Mutation;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.security.Authorizations;
import org.apache.hadoop.io.Text;

public class ReversingIntSortOrderWithMockInstanceDriver {

  static byte[] convert(int value) {
    byte[] key = ByteBuffer.allocate(4).putInt(value).array();
    byte[] reverse_key = ByteBuffer.allocate(4).putInt(Integer.MAX_VALUE - value).array();
    byte[] rv = new byte[8];
    for (int i = 0; i < 4; i++) {
      rv[i] = reverse_key[i];
    }
    for (int i = 0; i < 4; i++) {
      rv[i + 4] = key[i];
    }
    return rv;
  }

  public static String toHexString(byte[] ba) {
    StringBuilder str = new StringBuilder();
    for (int i = 0; i < ba.length; i++)
      str.append(String.format("%x", ba[i]));
    return str.toString();
  }

  public static void main(String[] args) throws IOException, AccumuloException, AccumuloSecurityException, TableExistsException, TableNotFoundException {
    Instance mock = new MockInstance("development");
    Connector connector = mock.getConnector("root", "password".getBytes());
    connector.tableOperations().create("TABLEA");

    BatchWriter wr = connector.createBatchWriter("TABLEA", 10000000, 10000, 5);
    for (int i = 5; i > 0; --i) {
      byte[] key = convert(i);
      Mutation m = new Mutation(new Text(key));
      m.put("cf", "cq", "value");
      wr.addMutation(m);
    }
    wr.close();

    Scanner scanner = connector.createScanner("TABLEA", new Authorizations());
    Iterator<Map.Entry<Key, Value>> iterator = scanner.iterator();
    while (iterator.hasNext()) {
      Map.Entry entry = iterator.next();
      Key key = entry.getKey();
      System.out.println(toHexString(key.getRow().getBytes()));
    }
  }
}

This code produces this output:

7ffffffa0005
7ffffffb0004
7ffffffc0003
7ffffffd0002
7ffffffe0001

I'll leave it to the reader to remove the debugging code.

How Can I Use Reverse Sort On Generic Accumulo Keys?

2012-11-12T16:02:00.002-05:00

This note shows how to reverse the sorting of Accumulo (actually, the row values). As you might know, the standard sort order is lexical. This first example shows a standard usage of a mock Accumulo instance. Notice that records inserted in reverse order (5, 4, 3, 2, 1) but are printed in lexical order.

public static void main(String[] args) throws Exception {
  // connect to a mock Accumulo instance.
  Instance mock = new MockInstance("development");
  Connector connector = mock.getConnector("root", "password".getBytes());
  connector.tableOperations().create("TABLEA");
  BatchWriter wr = connector.createBatchWriter("TABLEA", 10000000, 10000, 5);

  // insert five records in reverse order.
  for (int i = 5; i > 0; --i) {
    byte[] key = ("row_" + String.format("%04d", i)).getBytes();
    Mutation m = new Mutation(new Text(key));
    m.put("cf_" + String.format("%04d", i), "cq_" + 1, "val_" + 1);
    wr.addMutation(m);
  }
  wr.close();

  // display records; notice they are lexically sorted.
  Scanner scanner = connector.createScanner("TABLEA", new Authorizations());
  Iterator<Map.Entry&lyKey, Value>> iterator = scanner.iterator();
  while (iterator.hasNext()) {
    Map.Entry entry = iterator.next();
    Key key = entry.getKey();
    System.out.println("ROW ID: " + key.getRow());
  }
 }

The above code displays:

ROW ID: row_0001
ROW ID: row_0002
ROW ID: row_0003
ROW ID: row_0004
ROW ID: row_0005

Reverse sorting is accomplished by subtracting each byte in the row id from 255 as shown in the example below.

static byte[] convert(byte[] row) {
  byte[] rv = new byte[row.length * 2];
  for (int i = 0; i < row.length; i++) {
    rv[i] = (byte) (255 - row[i]);
  }
  for (int i = 0; i < row.length; i++) {
    rv[i + row.length] = row[i];
  }
  return rv;
 }

 public static void main(String[] args) throws Exception {
  // connect to a mock Accumulo instance.
  Instance mock = new MockInstance("development");
  Connector connector = mock.getConnector("root", "password".getBytes());
  connector.tableOperations().create("TABLEA");
  BatchWriter wr = connector.createBatchWriter("TABLEA", 10000000, 10000, 5);

  // insert five records in reverse order.
  for (int i = 5; i > 0; --i) {
    byte[] key = ("row_" + String.format("%04d", i)).getBytes();
    byte[] reverse_key = convert(key);
    Mutation m = new Mutation(new Text(reverse_key));
    m.put("cf_" + String.format("%04d", i), "cq_" + 1, "val_" + 1);
    wr.addMutation(m);
  }
  wr.close();

  // display records; notice they are lexically sorted.
  Scanner scanner = connector.createScanner("TABLEA", new Authorizations());
  Iterator<Map.Entry&lyKey, Value>> iterator = scanner.iterator();
  while (iterator.hasNext()) {
    Map.Entry entry = iterator.next();
    Key key = entry.getKey();
    System.out.println("ROW ID: " + key.getRow());
  }
 }

The above code displays:

ROW ID: ��������row_0005
ROW ID: ��������row_0004
ROW ID: ��������row_0003
ROW ID: ��������row_0002
ROW ID: ��������row_0001

It's important to note that for teaching purposes, the key is stored once in reverse format and again normally. Thus when displayed you can verify that the key is stored in reverse order. Normally the convert method is used like this:

static byte[] convert(byte[] row) {
  byte[] rv = new byte[row.length];
  for (int i = 0; i < row.length; i++) {
    rv[i] = (byte) (255 - row[i]);
  }
  return rv;
 }

For some use cases, you can convert the row bytes in place:

static byte[] convert(byte[] row) {
  for (int i = 0; i < row.length; i++) {
    row[i] = (byte) (255 - row[i]);
  }
  return row;
 }

Considerations for a Manual Partitioning Strategy in Accumulo

2012-10-30T16:59:00.001-04:00

On the Accumulo Users mailing list, Adam F. suggested:

1. Parallelism and balance at ingest time. You need to find a happy medium between too few partitions (not enough parallelism) and too many partitions (tablet server resource contention and inefficient indexes). Probably at least one partition per tablet server being actively written to is good, and you'll want to pre-split so they can be distributed evenly. Ten partitions per tablet server is probably not too many -- I wouldn't expect to see contention at that point.

2. Parallelism and balance at query time. At query time, you'll be selecting a subset of all of the partitions that you've ever ingested into. This subset should be bounded similarly to the concern addressed in #1, but the bounds could be looser depending on the types of queries you want to run. Lower latency queries would tend to favor only a few partitions per node.

3. Growth over time in partition size. Over time, you want partitions to be bounded to less than about 10-100GB. This has to do with limiting the maximum amount of time that a major compaction will take, and impacts availability and performance in the extreme cases. At the same time, you want partitions to be as large as possible so that their indexes are more efficient.

One strategy to optimize partition size would be to keep using each partition until it is "full", then make another partition. Another would be to allocate a certain number of partitions per day, and then only put data in those partitions during that day. These strategies are also elastic, and can be tweaked as the cluster grows.

In all of these cases, you will need a good load balancing strategy. The default strategy of evening up the number of partitions per tablet server is probably not sufficient, so you may need to write your own tablet load balancer that is aware of your partitioning strategy.

How Many Accumuo Loggers Are Needed Related To Slaves (tserver) Nodes?

2012-10-16T23:28:00.002-04:00

From the Accumulo User mailing list on Oct 10, 2012, John V says:

The scripts start loggers on all of the slave nodes. You can arbitrarily start/stop loggers, but we highly recommend doing 1:1 between them and tservers because the method for assigning tserver->logger does make assumptions depending on that 1:1 correlation.

How Does the Accumulo Client Find the Correct Tablet Server?

2012-10-16T23:26:00.000-04:00

From the Accumulo User mailing list on Oct 10, 2012, John V says:

Depending on the Instance your using, the client may hit HDFS first to get some instance information. Then either instance will hit Zookeeper to get the root tablet location. It will then scan that to find the appropriate !METADATA tablet, but will aggressively cache everything. The client will then scan (with heavy caching again) the !METADATA tablets necessary to find the tablet(s) for the range of writes. And then the writes will occur to appropriate tablets discovered in the !METADATA scan.

How To Run Multiple Instances of Accumulo on One Hadoop Cluster

2012-10-02T18:23:00.000-04:00

On the Accumulo User mailing list, Kristopher K. asked:

I built 1.5 from source last night and wanted to try it out on my existing Hadoop cluster without overwriting my current 1.4 set. Is there a way to specify the /accumulo directory in HDFS such that you can run multiple instances?

Eric N. replied:

From the monitoring user interface, see the Documentation link, then Configuration, see the first property:

instance.dfs.dir

You'll also change all the port numbers from the defaults. And there's a port number in conf/generic_logger.xml that points to the logging port on the monitor.

For example, here are some entries from my conf/accumulo-site.xml file:

<property>
<name>master.port.client</name>
<value>10010</value>
</property>

<property>
<name>tserver.port.client</name>
<value>10011</value>
</property>

<property>
<name>gc.port.client</name>
<value>10101</value>
</property>

<property>
<name>trace.port.client</name>
<value>10111</value>
</property>

<property>
<name>monitor.port.client</name>
<value>11111</value>
</property>

<property>
<name>monitor.port.log4j</name>
<value>1560</value>
</property>

And conf generic_logger.xml:


<appender name="N1" class="org.apache.log4j.net.SocketAppender">
<param name="remoteHost" value="${org.apache.accumulo.core.host.log}"/>
<param name="port" value="1560"/>
<param name="application" value="${org.apache.accumulo.core.application}:${org.apache.accumulo.core.ip.localhost.hostname}"/>
<param name="Threshold" value="WARN"/>
</appender>

How Accumulo Compresses Keys and Values

2012-10-02T18:02:00.001-04:00

From the Acccumulo User mailing list, Keith T said:

There are two levels of compression in Accumulo. First redundant
parts of the key are not stored. If the row in a key is the same as
the previous row, then its not stored again. The same is done for
columns and time stamps. After the relative encoding is done a block
of key values is then compressed with gzip.

As data is read from an RFile, when the row of a key is the same as
the previous key it will just point to the previous keys row. This is
carried forward over the wire. As keys are transferred, duplicate
fields in the key are not transferred.

General consensus seemed to favor double compression - compression both at the application level (i.e., compress the values) and let Accumulo compress as well (i.e., the relative encoding).

In support of double compression, Ameet K. said:

I've switched to double compression as per previous posts and
its working nicely. I see about 10-15% more compression over just
application level Value compression.

Stackscript for Accumulo (on Linode)

2012-08-28T22:09:00.000-04:00

I wanted a way to test the very latest Accumulo code. I could not use any of my existing systems because they were multi-use boxes. So I wrote a Stackscript in order to prepare a Linode server. Note that this script pulls a few files from my affy.com server for simplicity.

Users: hadoop, zookeeper, accumulo
Password: password

This stackscript downloads, installs and configures hadoop, zookeeper, and accumulo.

Step one is to create your own stackscript. Mine was called "InitializeAccumulo". The idea is that the starter script pulls the actual script from some server so that you don't need to deal with the 'Manage Stackscript' interface. The starter script is:

#!/bin/bash
wget http://www.affy.com/linode_build/stackscript -O /tmp/stackscript.sh
chmod +x /tmp/stackscript.sh
/tmp/stackscript.sh

After the Linode server boots, it will take about 10 minutes to run this script.

Get the latest copy of the stackscript from https://github.com/medined/accumulo_stackscript.

It's now possible to grab the whole project and then run the stackscript manually.

Resolving the "Permission Denied" Message On Accumulo Monitor Page

2012-07-23T19:11:00.000-04:00

If your operating system groups are not set correctly, you'll see a "Permission Denied" message under the NameNode section of the Accumulo Monitor page.

This message can be avoided by the following steps:

sudo groupadd supergroup
sudo useradd -G supergroup accumulo

Then restart Accumulo. These steps assume that your Accumulo processes are running under the accumulo user. If not, substitute the correct userid.

Resolving "./zookeeper.out: Permission denied" error

2012-07-23T18:25:00.000-04:00

When starting zookeeper, using the following command

sudo -u zookeeper $ZOOKEEPER_HOME/bin/zkServer.sh start

I saw this error:

Starting zookeeper ... /usr/local/zookeeper/bin/zkServer.sh: line 105: ./zookeeper.out: Permission denied

I resolved this error by telling zookeeper where I wanted the log file to be placed. Simply edit $ZOOKEEPER_HOME/zkEnv.sh, then add this assignment at the top of the file:

ZOO_LOG_DIR=/var/log/zookeeper

Then create that directory:

sudo mkdir /var/log/zookeeper
sudo chown zookeeper /var/log/zookeeper

How I Update To A New Version of Accumulo In Development

2012-07-12T23:53:00.001-04:00

There may be a better technique.

cd /usr/local
tar xvfz /tmp/accumulo-1.5.0-SNAPSHOT-dist.tar.gz
mv accumulo-1.5.0-SNAPSHOT accumulo-1.5.0-SNAPSHOT_`date +%Y_%m_%d`

# Copy the old configuration
cp $ACCUMULO_HOME/conf/* accumulo-1.5.0-SNAPSHOT_`date +%Y_%m_%d`/conf

chown -R accumulo:accumulo accumulo-1.5.0-SNAPSHOT_`date +%Y_%m_%d`

# Switch the symbolic link to the new directory.
ln -s accumulo-1.5.0-SNAPSHOT_`date +%Y_%m_%d` accumulo

sudo su - accumulo
cd $ACCUMULO_HOME
mkdir logs
mkdir walogs

bin/start-all.sh

Installing Accumulo on Linode

2012-07-07T13:48:00.000-04:00

I updated my github playground with steps to install Accumulo on a Linode server. The whole process, from buying the server to running the Accumulo shell, took less than 30 minutes. The steps are located at https://github.com/medined/accumulo_playground/blob/master/README_LINODE.txt.

Where Are the XSSF POI Classes?

2012-06-14T20:48:00.001-04:00

In the poi-ooxml jar file. This fact took me 20 minutes to find. Hopefully, you found this page faster.

Accumulo: Where is my instance id?

2012-06-11T21:51:00.000-04:00

When running a Java program to interact with Accumulo, I ran into the following exception:

Exception in thread "main" java.lang.RuntimeException: Accumulo not initialized, there is no instance id at /accumulo/instance_id
    at org.apache.accumulo.core.client.ZooKeeperInstance.getInstanceIDFromHdfs(ZooKeeperInstance.java:263)
    at org.apache.accumulo.server.client.HdfsZooInstance._getInstanceID(HdfsZooInstance.java:123)
    at org.apache.accumulo.server.client.HdfsZooInstance.getInstanceID(HdfsZooInstance.java:116)

/accumulo/instance_id should be an HDFS directory with just one file whose name is that of the Accumulo instance. For example:

$ hadoop fs -ls /accumulo/instance_id
Found 1 items
-rw-r--r--   1 medined supergroup          0 2012-06-11 17:58 /accumulo/instance_id/b519799c-3a51-4c9b-af21-96d577e2c11f

This issue was resolved when I added the Hadoop configuration directory to my classpath. Making this change enabled the Java program to look at HDFS instead of the local filesystem.