HuggingFace’s datasets project uses MMAP to make datasets available to multiple processes in an efficient way. This is very important since typically a machine learning training program will use a Dataloader which may use multiple workers, or alternatively the same dataset is simply accessed by multiple processes.

An issue was posted that suggested that a datasets-based program leaks memory with each iteration. This triggered an extensive research into understanding that MMAP doesn’t leak memory and bringing a lot of deepeer understanding of the different components used under the hood of datasets.

If you’d like to gain a deeper understanding into why and how please read on.

Emulating a computer with just 1GB of memory

Since we don’t want to crash our computer while debugging memory issues we are going to emulate a computer with just 1GB of memory and no swap memory. Unless such computer has a protection from programs using more memory than the computer has most of the time such computers start thrashing and eventually crash.

To accomplish that we are going to start a cgroups-controlled shell which will kill any program started from that shell and which consumes more than 1GB of memory (and give it no swap memory either):

$ systemd-run --user --scope -p MemoryHigh=1G -p MemoryMax=1G -p MemorySwapMax=0G --setenv="MEMLIMIT=1GB" bash

I’m setting MEMLIMIT=1GB env variable so that at any moment I can check if I’m in the right shell by printing:

$ echo $MEMLIMIT
1GB

Let’s validate that this shell allows a program to allocate under 1GB of RSS RAM, but will kill it if it tries to allocate more than that:

# 7 * 128M chars
$ python -c "import sys, os, psutil; a='a'*7*2**27; print(f'{psutil.Process(os.getpid()).memory_info().rss >> 20}MB');"
908MB

# 8 * 128M chars
$ python -c "import sys, os, psutil; a='a'*8*2**27; print(f'{psutil.Process(os.getpid()).memory_info().rss >> 20}MB');"
Killed

So we can see that < ~1GB works, but beyond an allocation that asks for more than 1GB of resident memory gets killed.

In the rest of this write up let’s use shell A, which is unlimited (or rather limited to an actual available memory on your computer) and shell B, where a program started from it can only allocate 1GB of resident memory.

Sidenote: Linux memory management and reporting is super-complicated and one could probably easily write a whole book about it. Resident Set Size (RSS) is typically the easiest to use to measure the approximate actual memory usage by the program. It doesn’t tell you the whole truth, but most of the time it’s good enough to detect memory leaks. Therefore in this write up this is the metric we are going to use.

Simple IO debug program

Now let’s write a simple debug program that will create a file with a few very large lines, and then it’ll read them sequentially using a normal IO, but if we set --mmap it’ll switch to memory mmaped API via the mmap module.

Additionally, if --accumulate flag is passed the program will accumulate the lines it reads into a single string.

$ cat python mmap-no-leak-debug.py
import gc
import mmap
import os
import psutil
import sys

PATH = "./tmp.txt"
# create a large data file with a few long lines
if not os.path.exists(PATH):
    with open(PATH, "w") as fh:
        s = 'a'* 2**27 + "\n" # 128MB
        # write ~2GB file
        for i in range(16):
            fh.write(s)

proc = psutil.Process(os.getpid())
def mem_read():
    gc.collect()
    return proc.memory_info().rss / 2**20

print(f"{'idx':>4} {'RSS':>10}   {'Δ RSS':>12}   {'Δ accumulated':>10}")

content = ''
mem_after = mem_before_acc = mem_after_acc = mem_before = proc.memory_info().rss / 2**20
print(f"{0:4d} {mem_after:10.2f}MB {mem_after - 0:10.2f}MB {0:10.2f}MB")

mmap_mode = True if "--mmap" in sys.argv else False

with open(PATH, "r") as fh:

    if mmap_mode:
        mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)

    idx = 0
    while True:
        idx += 1
        mem_before = mem_read()
        line = mm.readline() if mmap_mode else fh.readline()
        if not line:
            break

        if "--accumulate" in sys.argv:
            mem_before_acc = mem_read()
            content += str(line)
            mem_after_acc = mem_read()

        mem_after = mem_read()

        print(f"{idx:4d} {mem_after:10.2f}MB {mem_after - mem_before:10.2f}MB {mem_after_acc - mem_before_acc:10.2f}MB")

The four output columns are:

 idx        RSS          Δ RSS   Δ accumulated

1. the line number (starting from 1)
2. the total RSS reported at the end of each iteration
3. the RSS delta of each iteration
4. the accumulated buffer delta

And as you can see we force Python’s garbage collection via gc.collect() before taking RSS (Resident Set Size) measurements. This is a very crucial step when debugging memory usages and leaks in particular and especially if you delete some objects and want to make sure that memory is actually freed as Python’s garbage collection mechanism is not immediate.

Normal IO diagnostics

First, let’s run normal IO without accumulating any strings and simply discarding those.

shell A $ python mmap-no-leak-debug.py
 idx        RSS          Δ RSS   Δ accumulated
   0      12.37MB      12.37MB       0.00MB
   1     269.66MB     257.29MB       0.00MB
   2     269.68MB       0.02MB       0.00MB
   3     269.68MB       0.00MB       0.00MB
   4     269.69MB       0.01MB       0.00MB
   5     269.69MB       0.00MB       0.00MB
   6     269.70MB       0.01MB       0.00MB
   7     269.70MB       0.00MB       0.00MB
   8     269.70MB       0.01MB       0.00MB
   9     269.70MB       0.00MB       0.00MB
  10     269.71MB       0.01MB       0.00MB
  11     269.71MB       0.00MB       0.00MB
  12     269.71MB       0.00MB       0.00MB
  13     269.71MB       0.00MB       0.00MB
  14     269.71MB       0.00MB       0.00MB
  15     269.71MB       0.00MB       0.00MB
  16     145.96MB    -123.75MB       0.00MB

We read in a loop a 128MB line and discard it.

We can see the memory is very low and steady, with some fluctuations when Python decided to release some memory. The program allocates more than 128MB due to a new line character in the string – this is a peculiar Python behavior.

The bottom line is that the program doesn’t appear to be leaking any memory.

MMAP’ed IO diagnostics

Now let’s do the exact same operation but this time using mmap‘s IO:

shell A $ python mmap-no-leak-debug.py --mmap
idx        RSS          Δ RSS   Δ accumulated
   0      12.39MB      12.39MB       0.00MB
   1     268.25MB     255.87MB       0.00MB
   2     396.47MB     128.22MB       0.00MB
   3     524.47MB     128.00MB       0.00MB
   4     652.47MB     128.00MB       0.00MB
   5     780.47MB     128.00MB       0.00MB
   6     908.47MB     128.00MB       0.00MB
   7    1036.47MB     128.00MB       0.00MB
   8    1164.47MB     128.00MB       0.00MB
   9    1292.47MB     128.00MB       0.00MB
  10    1420.47MB     128.00MB       0.00MB
  11    1548.47MB     128.00MB       0.00MB
  12    1676.47MB     128.00MB       0.00MB
  13    1804.47MB     128.00MB       0.00MB
  14    1932.47MB     128.00MB       0.00MB
  15    2060.47MB     128.00MB       0.00MB
  16    2188.47MB     128.00MB       0.00MB

Whoah! It looks like there is a major leak here. On each iteration the program keeps on growing by 128MB despite us discarding the read data. What’s going on?

The theoretical explanation is simple – MMAP was designed to make IO faster and shared by multiple processes – so if there is a lot of available RAM, the MMAP API will use as much of it as it can and in order to speed things up it won’t normally release it back to the OS. For example, if you have two programs reading the same sections from the same MMAP’ed file only the first program will incur the delay of copying the data from disc to RAM. The other program will read it directly from RAM. Since MMAP doesn’t know which sections will be accessed next it simply keeps everything it read in the memory if there is enough of it.

But you’d say this is very bad and that’s a terrible design. But wait, it only keeps it in memory if nobody else wants the memory, and it immediately releases that unused memory back to the operating system as soon as such demand arises.

Proof that there is no leak

To show that the memory does get released as soon as it’s needed let’s re-run this same program in shell B, where only 1GB of memory is allowed to be allocated.

shell B $ systemd-run --user --scope -p MemoryHigh=1G -p MemoryMax=1G -p MemorySwapMax=0G --setenv="MEMLIMIT=1GB" bash
shell B $ python mmap-no-leak-debug.py --mmap
 idx        RSS          Δ RSS   Δ accumulated
   0      12.48MB      12.48MB       0.00MB
   1     268.51MB     256.03MB       0.00MB
   2     396.73MB     128.22MB       0.00MB
   3     524.73MB     128.00MB       0.00MB
   4     652.73MB     128.00MB       0.00MB
   5     780.73MB     128.00MB       0.00MB
   6     908.73MB     128.00MB       0.00MB
   7    1036.73MB     128.00MB       0.00MB
   8    1164.73MB     128.00MB       0.00MB
   9    1292.73MB     128.00MB       0.00MB
  10    1420.73MB     128.00MB       0.00MB
  11    1548.73MB     128.00MB       0.00MB
  12    1676.73MB     128.00MB       0.00MB
  13    1804.73MB     128.00MB       0.00MB
  14    1932.73MB     128.00MB       0.00MB
  15    2060.73MB     128.00MB       0.00MB
  16    2188.69MB     127.95MB       0.00MB

A surprise, it appears that the program managed to allocate >2GB of memory when we double checked that it should have been killed as soon as it reached 1GB RSS since we limited the shell to allow only <1GB memory allocation!

We will understand better shortly what’s going on, but it’s clear that cgroups that controls the memory usage is aware that while it accounts that MMAP’ed memory to the RSS counter of the program it’s aware that the program itself isn’t using most of this memory!

Interim observation: we can’t rely on RSS memory stats to diagnose memory leaks when MMAP is used.

Let’s create memory pressure

This is where our --accumulate flag comes in. It’s going to help us to see that RSS is “misreporting” the actual memory used by the program.

First we run it with normal IO:

shell A $ python mmap-no-leak-debug.py --accumulate
 idx        RSS          Δ RSS   Δ accumulated
   0      12.30MB      12.30MB       0.00MB
   1     269.60MB     257.29MB       0.00MB
   2     525.49MB     255.89MB     127.93MB
   3     653.49MB     128.00MB     127.87MB
   4     781.50MB     128.01MB     127.87MB
   5     909.50MB     128.00MB     127.87MB
   6    1037.51MB     128.01MB     127.87MB
   7    1165.51MB     128.00MB     127.87MB
   8    1293.52MB     128.01MB     127.87MB
   9    1421.52MB     128.00MB     127.87MB
  10    1549.53MB     128.01MB     127.87MB
  11    1677.53MB     128.00MB     127.87MB
  12    1805.53MB     128.00MB     127.87MB
  13    1933.53MB     128.00MB     127.87MB
  14    2061.53MB     128.00MB     127.87MB
  15    2189.53MB     128.00MB     127.87MB
  16    2193.78MB       4.25MB     127.87MB

where RSS reports correctly 128*16 ~= 2048MB and then some for the other bits of the program, but the ballpark matches.

Now let’s activate MMAP and re-run:

shell A $ python mmap-no-leak-debug.py --mmap --accumulate
 idx        RSS          Δ RSS   Δ accumulated
   0      12.37MB      12.37MB       0.00MB
   1     396.39MB     384.02MB     128.13MB
   2     652.48MB     256.09MB     128.00MB
   3     908.48MB     256.00MB     128.00MB
   4    1164.48MB     256.00MB     128.00MB
   5    1420.48MB     256.00MB     128.00MB
   6    1676.48MB     256.00MB     128.00MB
   7    1932.48MB     256.00MB     128.00MB
   8    2188.48MB     256.00MB     128.00MB
   9    2444.48MB     256.00MB     128.00MB
  10    2700.48MB     256.00MB     128.00MB
  11    2956.48MB     256.00MB     128.00MB
  12    3212.48MB     256.00MB     128.00MB
  13    3468.48MB     256.00MB     128.00MB
  14    3724.48MB     256.00MB     128.00MB
  15    3980.48MB     256.00MB     128.00MB
  16    4236.46MB     255.98MB     128.00MB

Here we can see that RSS reports 2x memory than it actually uses.

And now let’s create pressure using our 1GB-limited shell B and use normal IO with accumulation:

shell B $ systemd-run --user --scope -p MemoryHigh=1G -p MemoryMax=1G -p MemorySwapMax=0G --setenv="MEMLIMIT=1GB" bash
shell B $ python mmap-no-leak-debug.py --accumulate
 idx        RSS          Δ RSS   Δ accumulated
   0      12.38MB      12.38MB       0.00MB
   1     269.41MB     257.04MB       0.00MB
   2     525.55MB     256.14MB     127.93MB
   3     653.55MB     128.00MB     127.87MB
   4     781.56MB     128.01MB     127.87MB
   5     909.56MB     128.00MB     127.87MB
Killed

As you can easily see the program gets killed once it reaches 1GB of RSS. It managed to perform 5 iterations, thus on iteration 6 it tries to accumulate 6*128=768 plus the current readline read of 128MB, plus the memory used by the rest of the program, it crosses 1GB and gets killed before finishing iteration 6.

Also it might be useful to compare with the same run with shell A. You can see that RSS of the shell B run is quite different from shell A. The reported RSS doesn’t grow as fast.

Now let’s run the MMAPed version:

shell B $ systemd-run --user --scope -p MemoryHigh=1G -p MemoryMax=1G -p MemorySwapMax=0G --setenv="MEMLIMIT=1GB" bash
shell B $ python mmap-no-leak-debug.py --mmap --accumulate
 idx        RSS          Δ RSS   Δ accumulated
   0      12.51MB      12.51MB       0.00MB
   1     396.52MB     384.00MB     128.13MB
   2     652.60MB     256.08MB     128.00MB
   3     908.60MB     256.00MB     128.00MB
   4    1164.60MB     256.00MB     128.00MB
   5    1420.60MB     256.00MB     128.00MB
Killed

You can see it gets killed in the exactly same iteration as when it was run without MMAP.

You can see that while the RSS numbers are bigger than that of the normal IO run, the program gets killed in the exact same iteration. which tells us the actual memory usage with normal IO and mmap’ed IO is either very similar or very likely exactly the same.

What about PyArrow?

Originally this whole research started from this Issue in the datasets repo. It looked like a dataset loaded via pyarrow leaked on every iteration.

Quentin Lhoest reduced it to a simple pyarrow program

$ cat mmap-no-leak-debug-pyarrow.py
import psutil
import os
import gc
import pyarrow as pa

ARROW_PATH = "tmp.arrow"

if not os.path.exists(ARROW_PATH):
    arr = pa.array([b"a" * (200 * 1024)] * 1000)  # ~200MB
    table = pa.table({"a": arr})

    with open(ARROW_PATH, "wb") as f:
        writer = pa.RecordBatchStreamWriter(f, schema=table.schema)
        writer.write_table(table)
        writer.close()

def memory_mapped_arrow_table_from_file(filename: str) -> pa.Table:
    memory_mapped_stream = pa.memory_map(filename)
    opened_stream = pa.ipc.open_stream(memory_mapped_stream)
    pa_table = opened_stream.read_all()
    return pa_table


table = memory_mapped_arrow_table_from_file(ARROW_PATH)
arr = table[0]

print(f"{'idx':>8} {'RSS':>10} {'Δ RSS':>15}")

mem_before = psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024)
for idx, x in enumerate(arr):
    if idx % 100 == 0:
        gc.collect()
        mem_after = psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024)
        print(f"{idx:4d}  {mem_after:12.4f}MB {mem_after - mem_before:12.4f}MB")

which when run produced the familiar leak-like pattern:

$ python mmap-no-leak-debug-pyarrow.py
     idx        RSS           Δ RSS
   0       51.3164MB       2.5430MB
 100       69.9805MB      21.2070MB
 200       90.6055MB      41.8320MB
 300      107.1055MB      58.3320MB
 400      127.7305MB      78.9570MB
 500      148.3555MB      99.5820MB
 600      164.8555MB     116.0820MB
 700      185.4805MB     136.7070MB
 800      206.1055MB     157.3320MB
 900      226.7305MB     177.9570MB

But if we run it from a shell that is only allowed 100MB of allocated memory:

$ systemd-run --user --scope -p MemoryHigh=0.1G -p MemoryMax=0.1G -p MemorySwapMax=0G --setenv="MEMLIMIT=0.1GB" bash
$ python mmap-no-leak-debug-pyarrow.py
     idx        RSS           Δ RSS
   0       51.2852MB       2.4609MB
 100       70.4102MB      21.5859MB
 200       86.9102MB      38.0859MB
 300      107.5352MB      58.7109MB
 400      128.1602MB      79.3359MB
 500      148.7852MB      99.9609MB
 600      165.2852MB     116.4609MB
 700      185.9102MB     137.0859MB
 800      206.5352MB     157.7109MB
 900      227.1602MB     178.3359MB

So it reports it allocated ~200MB of RSS, yet it runs just fine without getting killed.

There is no leak here.

What about HuggingFace datasets?

In another Issue a very similar datasets-iterator-is-leaking report was submitted.

So let’s use a similar datasets reproduction example here but we will use a larger dataset.

$ cat mmap-no-leak-debug-datasets.py
from datasets import load_dataset
import gc
import os
import psutil
import sys

keep_in_memory = True if "in-mem" in sys.argv else False

proc = psutil.Process(os.getpid())
def mem_read():
    gc.collect()
    return proc.memory_info().rss / 2**20

dataset = load_dataset("wmt19", 'cs-en', keep_in_memory=keep_in_memory, streaming=keep_in_memory)['train']
print(f"Dataset len={len(dataset)}")

print(f"{'idx':>8} {'RSS':>10} {'Δ RSS':>15}")
step = 1_000_000
mem_start = 0
for idx, i in enumerate(range(0, len(dataset), step)):
    if idx == 4: # skip the first few iterations while things get set up
        mem_start = mem_read()
    mem_before = mem_read()
    x = dataset[i:i+step]
    mem_after = mem_read()
    print(f"{idx:8d} {mem_after:12.4f}MB {mem_after - mem_before:12.4f}MB ")
mem_end = mem_read()

print(f"Total diff: {mem_end - mem_start:12.4f}MB ")

Let’s run it in a normal shell first:

$ python mmap-no-leak-debug-datasets.py
Dataset len=7270695
     idx        RSS           Δ RSS
       0     775.7773MB     609.9805MB
       1     849.6016MB      73.8242MB
       2     876.1445MB      26.5430MB
       3     941.3477MB      65.2031MB
       4     984.9570MB      43.6094MB
       5    1053.6445MB      68.6875MB
       6    1164.2852MB     110.6406MB
       7    1252.5312MB      88.2461MB
       8    1368.6523MB     116.1211MB
       9    1445.7266MB      77.0742MB
      10    1564.5195MB     118.7930MB
      11    1678.7500MB     114.2305MB
      12    1729.9844MB      51.2344MB
      13    1866.1953MB     136.2109MB
Total diff:    1700.3984MB

You can see the mid-column of total RSS memory keeps on growing in MBs. The last column is by how much it has grown during a single iteration of the script (0.5M items).

And now let’s run in a 1GB limited shell:

$ systemd-run --user --scope -p MemoryHigh=1G -p MemoryMax=1G -p MemorySwapMax=0G --setenv="MEMLIMIT=1GB" bash
$ python mmap-no-leak-debug-datasets.py
Dataset len=7270695
     idx        RSS           Δ RSS
       0     775.8516MB     610.1797MB
       1     849.5820MB      73.7305MB
       2     876.1328MB      26.5508MB
       3     941.3281MB      65.1953MB
       4     984.9375MB      43.6094MB
       5    1053.6328MB      68.6953MB
       6    1164.0273MB     110.3945MB
       7    1252.5273MB      88.5000MB
       8    1368.3906MB     115.8633MB
       9    1445.7188MB      77.3281MB
      10    1564.2656MB     118.5469MB
      11    1678.7383MB     114.4727MB
      12    1729.7227MB      50.9844MB
      13    1866.1875MB     136.4648MB
Total diff:    1700.5156MB

No problem at all.

So we now know there is no leak there and it’s just the OS includes in RSS memory that will be released as soon as it’s needed.

How to debug real memory leaks while using MMAP

So how does one debug an actual memory that might be elsewhere in the code while using MMAP.

Well, you have to disable MMAP for the duration of your debug session and then re-enabled it back when you want high performance.

As you have seen at the beginning of this article switching from mmap to normal IO is very simple to do.

In the case of datasets you’d turn MMAP functionality off with keep_in_memory=True as in:

dataset = load_dataset("wmt19", 'cs-en', keep_in_memory=True, streaming=False)['train']

This loads the dataset in RAM, and now you should be able to debug your potential leak.

Let’s test after modifying our last program:

- dataset = load_dataset("wmt19", 'cs-en', keep_in_memory=False, streaming=False)['train']
+ dataset = load_dataset("wmt19", 'cs-en', keep_in_memory=True, streaming=False)['train']

Now in the normal unlimited shell we run:

$ python mmap-no-leak-debug-datasets.py --in-mem
Dataset len=7270695
     idx        RSS           Δ RSS
       0    1849.5391MB     469.5781MB
       1    1833.0391MB     -16.5000MB
       2    1803.4609MB     -29.5781MB
       3    1811.5312MB       8.0703MB
       4    1803.9531MB      -7.5781MB
       5    1811.7734MB       7.8203MB
       6    1836.0391MB      24.2656MB
       7    1839.5938MB       3.5547MB
       8    1855.9688MB      16.3750MB
       9    1850.5430MB      -5.4258MB
      10    1865.3398MB      14.7969MB
      11    1876.2461MB      10.9062MB
      12    1853.0469MB     -23.1992MB
      13    1881.4453MB      28.3984MB
Total diff:     501.4844MB

The RSS memory is more stable but fluctuates because the records are different, and the dataset can be huge to load into memory.

Using synthetic MMAP-disabled dataset to debug memory leaks

Therefore the easiest approach is to create a synthetic dataset of desired length with all records being the same. That way the data is no longer a factor in the memory usage patterns as it’s always the same.

$ cat ds-synthetic-no-mmap.py
from datasets import load_from_disk, Dataset
import gc
import sys
import os
import psutil

proc = psutil.Process(os.getpid())
def mem_read():
    gc.collect()
    return proc.memory_info().rss / 2**20

DS_PATH = "synthetic-ds"
if not os.path.exists(DS_PATH):
    records = 1_000_000
    print("Creating a synthetic dataset")
    row = dict(foo=[dict(a='a'*500, b='b'*1000)])
    ds = Dataset.from_dict({k: [v] * records for k, v in row.items()})
    ds.save_to_disk(DS_PATH)
    print("Done. Please restart the program")
    sys.exit()

dataset = load_from_disk(DS_PATH, keep_in_memory=True)
print(f"Dataset len={len(dataset)}")

print(f"{'idx':>8} {'RSS':>10} {'Δ RSS':>15}")
mem_start = 0
step = 50_000
warmup_iterations = 4
for idx, i in enumerate(range(0, len(dataset), step)):
    if idx == warmup_iterations: # skip the first few iterations while things get set up
        mem_start = mem_read()
    mem_before = mem_read()
    _ = dataset[i:i+step]
    mem_after = mem_read()
    print(f"{i:8d} {mem_after:12.4f}MB {mem_after - mem_before:12.4f}MB")
mem_end = mem_read()

print(f"Total diff: {mem_end - mem_start:12.4f}MB (after {warmup_iterations} warmup iterations)")

We run this program once to create the dataset, and then the second time to profile its memory usage:

$ python ds-synthetic-no-mmap.py
Creating a synthetic dataset
Done. Please restart the program

$ python ds-synthetic-no-mmap.py
Dataset len=1000000
     idx        RSS           Δ RSS
       0    1649.6055MB      95.1992MB
   50000    1728.4961MB      78.8906MB
  100000    1728.7109MB       0.2148MB
  150000    1729.2539MB       0.5430MB
  200000    1729.0039MB      -0.2500MB
  250000    1729.5039MB       0.5000MB
  300000    1729.2539MB      -0.2500MB
  350000    1729.7539MB       0.5000MB
  400000    1729.5039MB      -0.2500MB
  450000    1730.0039MB       0.5000MB
  500000    1729.7539MB      -0.2500MB
  550000    1730.2539MB       0.5000MB
  600000    1730.0039MB      -0.2500MB
  650000    1730.5039MB       0.5000MB
  700000    1730.2539MB      -0.2500MB
  750000    1730.7539MB       0.5000MB
  800000    1730.5039MB      -0.2500MB
  850000    1731.0039MB       0.5000MB
  900000    1730.7539MB      -0.2500MB
  950000    1731.2539MB       0.5000MB
Total diff:       2.0000MB (after 4 warmup iterations)

This is much better. There are still tiny fluctuations due to Python and you can see in the code I skipped the first few iterations in the code while things are being set up.

But otherwise now you can easily debug the rest of your code for any memory leaks since datasets are in non-MMAP mode and the records size doesn’t fluctuate.

Of course, do not forget to flip load_from_disk(..., keep_in_memory=True) to False when the debugging process is over so that you get back the performance speed up provided by MMAP.

I wrote these notes mainly for myself to ensure I have a good understanding of this complex use-case. And I hope you have gained some understanding from it as well.

The first stage was to get the card recognized by BIOS and then the OS which took quite some figuring out.

The second stage was to get a custom water cooling solution, since A100 runs really hot and it’s very difficult to cool with just fans and typically requires a server-level cooling hardware and a server room with air conditioning.

But after a few months of trials and tribulations I have a working solution, which I’ll share with you in this post.

Getting A100 recognized by BIOS

A100 PCIe is a headless card designed for server use – it has no sockets to plug a monitor in.

I first made a post at NVIDIA forums asking for help and another user found that the only way to make A100 work is to have another NVIDIA GPU to run the monitor from and that it’s currently impossible to use iGPU (built-in gpu available with most Intel CPUs consumer level motherboards).

To benefit from PCIe-4 I purchased ROG Maximus XIII Hero (z590) as my original MB was z390 / PCIe-3.

I first tried using iGPU (CPU Graphics) and A100 lead to the system not POSTING (d4 – PCI resource allocation error. Out of Resources).

As kindly shared by one of the forum users that the current solution is to use a 2nd card, I added a old NVIDIA PCIe card and plugged the monitor into it. Now it POSTed and booted just fine, but still not seeing A100 wasn’t visible in nvidia-smi.

I also tried changing the order of the cards (A100 2nd) – but there was no change in the outcome.

I then experimented with various BIOS configurations until I found one that worked. Here it is:

Advanced:

  Advanced System Agent (SA) configuration

    Graphics Configuration:
      Primary Display: Auto (probably could be set to PEG)
      IGPU Multi-Monitor: Disabled

    Memory Configuration:
      Memory Remap: Enabled (above 4GB)

    PCI Subsystem Settings
      Above 4G Decoding: Enabled
      Resize Bar: Enabled
      SR-IOV Support: Enabled

and the reason it wasn’t working originally is because by default it had SR-IOV Support: Disabled

As it’s possible that other motherboards may or may not work please read this thread where Scott Ellis explains which BIOS settings the server motherboards normally need to detect A100, and which may or may not be present in the consumer motherboard. Another user shared that they got it to work on ASUS ROG STRIX Z690-G GAMING WIFI, so chances are that the recent motherboards support it. If you find others that work please don’t hesitate to share the name in the comments.

Water Cooling A100

Fans proved to be very inadequate for A100 cooling, the huge radiator it came with did well for the first few minutes and then it’d remain really hot – definitely not something usable for the weak PC fans. Perhaps it’d work if I were to blast them at 100% speed but it’d be very loud and won’t work in summer.

I started researching water cooling.

I first ordered the water block from EKWB – they unfortunately didn’t label their product correctly – only in a small print it said that it was for A100 40GB which doesn’t fit A100 80GB PCB. So I wasted a lot of time and lost money on dealing with the wrong product. I asked them to fix the label to state that it’s a 40GB block, but they refuse to do it. And as of this writing they have no plans to make a 80GB version.

Here is the 40GB water block from EKWB:

But A100 80Gb added a metal frame around the main chip as can be seen below. It wasn’t there in the 40GB version of PCB.

Then I ordered Bykski’s A100 80GB water block via aliexpress. The ordering experience wasn’t great as I paid top dollar for quick shipping (DHL) and the vendor abused that, by submitting a shipment label but didn’t ship the product for weeks! As soon as I was able to start a dispute and request a full refused I did that, and immediately the product was shipped. So be careful spending extra money on shipment since Aliexpress allows its vendors to do what they please and they don’t enforce anything.

Here is the water block (left) and the PCB (right):

Taking apart A100 was mainly a matter of removing all the screws. The key to be able to remove the panel is to remove 2 screws around the power plug.

The first step is to clean the main chip from the thermal paste using alcohol and a coffee filter so that the old residue is removed.

Then thermal pads need to be applied. The instructions provided by Bykski product page are negligently incomplete. Luckily I still had the original radiator to be the model to where I need to apply the thermal pads:

You can see I matched them (actually I now see I missed one of them on the left! ouch)

Now it’s time to put the water block on and tighten the 4 screws around the main chip:

Bykski slacked again and didn’t provide a proper mounting bracket that would fit their product. Their instructions allude to using the bracket from the original A100 radiator. As you can see it had 4 screws mounting the main radiator body and 2 more on the side you can’t see on the picture:

so when you try to mount it on Bykski block you end up with it mostly hanging in the air and attached to the backplate with 2 super tiny screws. This is very poor engineering.

The acrylic block is super-heavy, I’m very concerned that this can break the PCIe socket or the A100 PCB.

(If Bykski engineers read this, please provide your own custom mounting bracket that gets screwed onto the acrylic block and perhaps the backplate. Otherwise your product can easily break the motherboard socket, since the original mounting bracket that comes with A100 was not designed for your watering block.)

Finally I put it all together with a radiator and a reservoir-pump combo from EKWB and had it run for 24h to test for leaks.

This was my first time doing a custom water cooling solution so it wasn’t easy. As I couldn’t afford to have my desktop not working for 24h, I did the whole water cooling ensemble outside and simply plugged it into an old computer while testing for leaks. And the next day I mounted the ensemble in the target PC while keeping the 3 parts interconnected, which was a bit of a challenge but I made it work.

I purchased the huge 7000D AIRFLOW Full-Tower ATX PC Case from Corsair and even then I had a hard time putting this huge EK-CoolStream CE 420 radiator together with the smaller AIO radiator I was using for CPU already. I have multiple HDs and the Corsair HX1200 power-supply that is extra long, so I barely had any space to place all those parts around.

Finally, I had to turn my headless solution for igpu and nvidia cards off, so that it could switch to the NVIDIA driver, while having the monitor plugged into an old GTX-1080 which was perfect as it is low power and doesn’t overheat as easily as the Ampere GPUs. And booted my Kubuntu.

And voila putting some serious load on the card it keeps a nice ~30C – Amazing!

So despite not thinking it fully through the Bykski’s water block works well at the moment.
And yes, A100 80Gb is 10x larger than GTX-1080.

Things to figure out

1. setting up software to trigger PWM and water pump adjustments based on temperature reported by nvidia-smi – at the moment I have just set a normal pump speed in BIOS and using the CPU’s AIO to drive the speed of the fans – typically when GPU is churning CPU is almost always busy as well.
2. Of course, figuring out how to get rid of the 2nd NVIDIA card and use iGPU instead. Save electric bills and generate less heat. If you discover a solution please share in the comments. Thank you!

Notes

This post is focused on the specifics of getting A100 80GB PCIe working in a PC, and I’m not an expert in water cooling, so besides sharing how I installed the water block itself, I trust you can find the details on the best way to do the water cooling elsewhere.

Huge thanks to the person on the NVIDIA forums who discovered the workaround by using a 2nd card to recognize A100.

Other related gpu-center-at-home tinkering projects

– A 4x A100-SXM setup at home by Leo Paille

The big news nowadays is Machine Learning (ML). But what is it? The machine part is obvious – it’s the computer that does the learning. But the learning part may appear as tricky, even though in reality it’s a very simple thing in its foundation. Let’s gain an intuitive understand of the learning in ML.

Let’s play a game

I don’t know if you have ever played a computer game where you have a ball at a starting position, and you have to direct it to a target with the help of blocks, springs, rotating gears, etc. The original game was named The Incredible Machine. It usually relies on the laws of physics and you need to re-arrange those helper components to make the ball move from let’s say the left top corner of the screen to the right bottom of the screen where some sort of a target is located. The way you play the game is by experimenting with the block arrangement, so that they will do the work of guiding the ball to its target.

Here is a possible initial game board setup:

The purple ball needs to hit the purple hexagon for you to win.

Normally, with the help of gravity, the ball just falls straight down. But since we have the helper blocks, we will use them to guide the ball so that once it starts falling it’ll have a trajectory to hit the target.

Now, I hope that it will hit the target.

But no, I slightly misplaced the “plank”-shaped block and the ball will get stuck in the gap.

OK, let’s move the bar a bit to the left:

Hooray, this time, the ball should hit the target.

Next, the initial ball position is moved a bit to the left.

And our helper block arrangement is no longer helping to guide the ball to the target. The ball again falls straight down.

OK, let’s rearrange some blocks:

And success! Now this starting position of the ball is covered too.

But the previous one (the ball starting from a different position) will no longer work. So, we have more trial and error re-arrangements to do.

After some lengthy experimentation we find the following arrangement:

And, voila, almost any starting position of the ball will now lead it to the target correctly.

Let’s revisit the stages of the game we have just played.

1. The first time I tried to arrange the blocks, I under-shot and missed the target. (You can’t tell from the picture how heavy the ball is or how much friction the helper blocks generate, so while it is looking obvious in my drawing, in the actual game it could be quite challenging to find the right setup).
2. Then I started experimenting by moving the helper blocks a bit and re-running the ball, continually making small corrections until the target was reached.
3. Then a new situation was given and I through trial and error I made it work.
4. Then I tried to arrange the blocks so that both situations would work.
5. Finally, I tried to generalize so that numerous possible situations would work.

Machine Learning

If you understood how the game was played, you already understands the learning part in the Machine Learning. This is how the ML model learns. It usually starts with a random arrangement of the helper blocks (hidden weights) and tests whether it hits the target (e.g. a specific category). If it didn’t succeed, it tries to make small adjustments, while measuring whether the ball is coming closer to the target or not (often using the method of gradient descent). Overtime, the adjustments become better and better, until it actually hits the target. It is possible for the model to overshoot the target, in which case it turns around and takes a few small steps in the opposite direction until the target is hit. This whole process is called training.

The state where the ball hasn’t made to the target for most ball starting positions yet, is called underfitting, since it can clearly still improve.

The more different positions we train the model with, the better it will generalize for positions it hasn’t yet seen. Assuming that we have an infinite number of helper blocks of different shapes we can probably find a solution for almost every situation.

If we try to fit perfectly all the situations we have seen so far, we may end up with a perfect outcome for every seen situation, but if a new situation is given it may not work. This is called overfitting.

And the final stage of the game was to make our setup generalize, so that it could fit almost any unseen situation. In practice, the generalization is an ongoing process, rather than a final stage. For example, by randomly removing the helper blocks at random situations, we can force our block arrangement to be more robust to unseen situations. This is called dropout in ML lingo.

Usually, we don’t have an infinite number of helper blocks and so we have to make do with what we have. So the number of block arrangement possibilities is limited. And, say, after 10 different ball position starting points, our model knows how to send the ball to its destination 90% of the time, which is already quite excellent.

Now you no longer need to spend hours moving the blocks around, since the computer will do it for you. This is no longer fun, since it’s the computer that’s now playing the game for you. But for the sake of better understanding of how Machine Learning works we will allow it to do it for us this time.

Deep Learning

Next comes the concept of Deep Learning, where you give the ML model a lot more helper blocks, allowing it to build much more elaborate setups, which usually we don’t even understand how they were made to work. And then the process is repeated with hundreds and thousands of different starting combinations and the model learns with, often, close to 100% correct solutions.

The deep part just indicates that instead of a simple basic setup with a few blocks, now we have a much more complex setup that has a much higher capacity for flexibility of arranging things and generalizing. For example, for the sake of our mechanical universe example, imagine that re-arrange our helper blocks so that they represent a concave surface like a bowl or a wine glass, and the target is at the bottom of it. If this was possible, no matter where the ball is released from it would always ends up at the bottom of that concave surface where our target is situated.

If you’re ready to imagine even more complex situations, consider multiple dimensions, so if it looks impossible to make the ball move from say the bottom of the board to the top, we use a forth dimension to sneak it in. In reality deep learning uses hundreds and thousand of dimensions to solve very tricky set ups. We can hardly visualize the 4th dimension, so we just have to trust that it works, relying primarily on math.

(note: If you would like to stretch your mind in a relatively easy and fun way to start grasping multiple dimensions read the book Flatland or watch the movie based on this book.)

Further, to make things even more efficient, instead of trying to figure out one ball situation at a time, the model tries to work simultaneously on batches of dozens or hundreds of such situations at once. This is due to availability of specialized hardware (called GPU or TPU), which was designed to process huge amounts of data in parallel at an incredible speed. Not only that hardware makes things run much faster, it also finds the best generalized block arrangement in less steps, as compared to doing it one situation at a time.

While the foundations of ML are very simple, for it work successfully and in a timely manner we need:

1. Either huge amount of data to train it on – think real estate prices for the last 5 years over a huge territory, which is often needed for supervised learning, where the data helps the model to improve. Or the model can be trained using reinforcement learning, like in our ball to target game example, where through trial and error it learns how to do better over time.
2. Very powerful hardware, usually specially designed for heavy matrix processing, to support deep learning. Imagine, how much more complicated it would be for you to play this game if you had to accomplish the same not with a few helper blocks, but millions of blocks.

If these two requirements are satisfied then Deep Learning is possible. The remaining difficult part is usually to build or choose an architecture that solves a problem at hand.

Transfer Learning

Now, if you were to start playing the arrange-the-blocks game from scratch you’d probably have to invest as much work as I did to make it deal with multiple starting situations. What if I were to save my work and share it with you? Then you won’t have to start from scratch and will be able to continue working on more complex situations, saving yourself a huge amount of time and computer resources.

This brings us to Transfer Learning, where an individual or an organization invests a huge amount of their time and computer resources (i.e. money) to build a ML model, which can then be shared with others.

This shared model is then fine-tuned to a specific type of data. For example imagine you want to replace the falling ball with some kind of polygon, that can roll like a ball, but it doesn’t roll as well. You will still benefit from the pre-trained model, and then you will make small adjustments to the block arrangement by retraining the existing model that was shared with you for your specific needs.

Often, transfer learning can provide huge savings of time and money.

Outcome Explainability

Deep Learning models often figure out very subtle correlations between input signals, that we humans are either not likely to notice or have the capacity to do so, but since ML models are forced to generalize to give correct answers, at times the answers are given for totally wrong or unexplainable reasons.

There is the urban legend, that once upon a time a certain military force tried to build a ML model to detect camouflaged tanks in satellite imagery. A ML model was trained with a handful of photos that included tanks (positive examples) and about the same amount with no tanks in them (negative examples). After training on these photos, the model was able to classify new photos correctly, except instead of learning to tell hidden tanks, it learned to tell whether it was a cloudy day or not. Since it so happened that all camouflaged tanks photos were taken on the cloudy days, and all the non-tank photos were taken on non-cloudy days.

I found an article that attempts to find the truth about whether there is any truth to this legend. I will let you discover it for yourself, but the article concludes with:

So I think it’s very likely, though not certain, that this didn’t actually happen.

Regardless, there are plenty of research papers out there that do indicate real findings of this type and this a big problem, since if we don’t know how a model makes its decisions, we will be unable to use it reliably. If some kitten photos gets miscategorized, it’s probably not a big deal, but if a self-driving car miscategorizes an obstacle and we can’t figure out why, or a person gets put in jail due to a ML model mistake, that would be a big problem.

Conclusion

I hope this little playful introduction helped you to gain an insight into how the very complicated field of Machine Learning is based on rather simple things. The complexity is in the details of finding the right solution for the right situation, knowing how to process data, how to debug problems in the code, optimize the code to work faster, etc. The devil is in the detail. But I trust you now have an intuitive understanding of how the gears of Machine Learning work and in particular the Learning part of it.

Unfortunately, this doesn’t work, because those who run social networks have a conflicting interest with yours.

Welcome to the Social Network Black Hole

Social networks primarily make income from advertising. In order to be able to sell advertisement placements, the social networks need a lot of eyeballs and that requires minions to generate new content all the time. Who are those minions? Well, it’s most of us…

Have you ever tried to find a post written by you or someone else from a few weeks earlier and you just couldn’t find it? Well, it’s because each piece of user-generated content has a very short life-span on social networks. It lives slightly longer if it generates a controversy and/or gets re-shared by many. But even that content disappears in a relatively short period of time. This is by design.

Even though, the older content is usually just as solid and relevant for years to come (a delicious shakshuka recipe anybody?), if all your followers have already seen that post, they won’t be excited to look at it and re-share it again, or start a new controversy, and therefore, it will generate no fresh eyeballs to display ads to. As such, the old content is pushed down to make space for a fresh, potentially exciting, content and thus to generate ad revenue for the social network owners.

Now, I hope you realize that your hard work makes money for the social networks. To which you may retort, but they sends me customers (if you’re trying to use social networks to create a customer flow that is). And I’d agree partially, and also tell you that this is an extremely inefficient way of accomplishing your goals. Let’s explore why this is the case.

If, tomorrow, you get sick and stop posting on your favorite social network, how long will it take before your customer flow from that network will come to a full halt? Usually, in 7 to 10 days of you “not contributing”, as far as the social network algorithms are concerned, you never existed, i.e. nobody will see all the hundreds or thousands of the earlier posts you worked so hard to create.

At least, if the social networks were to share profits with the content creators, but most don’t do that. The networks collect all the profits and you work for free.

Is there a way out? Yes, there is. You need to take the control back. You need to own all the content that you generate and have it hosted on your own website. You can pay $5/month or even have a free blog/website – just search for those and there are hundreds of great offers out there. For example many use free blogs like wordpress and blogger, but there are many other similar and even better options out there.

Some 10-15 years ago having your own website required either a hardcore tech expertise or one had to pay a ton to someone to build and maintain it. These days, even my grandma could probably learn how to write her own blog. If you can create a Word document, you can, with the same easy, write a blog. You can always create a free blog, experiment with it – it will take 1h of your time to get started – and chances are, you will find out that it’s a very easy task. It won’t take longer than, say, switching from an older MS Word to the Windows 10 version of the same application, and figure out the new interface.

I recommend investing the extra $10-15/year and getting your own domain. That way you may choose to move from one hosting company to another, but all your previously existing links will still work. If, however, you use a free domain like mygreatblog.wordpress.com, and then you decide you want to use a different platform, and now have mygreatblog.blogger.com, the old links to the original website will no longer work and the search engines will have to discover your website anew, meaning you will lose all that free traffic and you will need to start acquiring it from scratch.

If this article made you think and you decided to take control over your writing, start creating useful content, one paragraph or article at a time. Every time you create a new content on your website, do post a link to it to your social network. But you retain control over how, when and to whom your content is shown. And here are some other great benefits:

1. Now, your content, instead of having a life-span of days, will have a life-span of years.
2. Little by little search engines will start sending to you free traffic. And this will grow exponentially over time.
3. You can also buy ads on other sites to send more traffic to your own content.
4. If the content is interesting and unique, other sites and even social networks will start sending free traffic to you. wikipedia-style reference-type writing will guarantee a ton of traffic.
5. You can make landing pages if you are into selling goods and services.
6. You can publish content that otherwise won’t be allowed to be published on social networks.
7. You own and control the comments. For example you can delete comments you don’t approve of and edit comments with bad grammar.
8. You can revisit and enhance your content over time.
9. You can even make money from your content by selling ad space on your own website. This gives you passive income.

And while you may not be able to live the The 4-Hour Workweek, you should be able to take a break from producing any time, and for example take a month off and go on a long well-deserved and badly needed vacation. Your website/blog will continue working for you 24/7, despite you being on some remote island with no Internet connection.

I was recently browsing Google Analytics for some of my really old sites and I discovered that there are some articles I published 20 years ago and they are still being read today. For example in January 2000 I published a summary Employment and Stock Options Plan Explained, and 20 years later it’s still being read.

Note that this is a long term investment. You just need a bit of patience, as it will take some time for your new website/blog to be discovered by search engines. Meanwhile sending traffic to your website via social networks is quick and easy and there are even tools that can do this automatically for you. But, you no longer need to slave for a company that gives you very little in return and forgets about you as soon as you stopped giving.

Another important subject matter to explore is measuring outcomes. As, Dan Kennedy, the most insightful practical marketing guru that I have ever encountered, says – if you can’t measure the outcome of something you’re putting an effort into – do not do it. Because if you do, you have no idea whether your effort/money amounted to anything good, and you have no way of making improvements, since you don’t have a base line to compare with. You could be getting your customers anyway, and most of your social network effort could be mostly wasted. I think this is the case for many entrepreneurs, who try to get the social networks to work for them, but who can’t measure the real impact of their efforts. But, this is a big topic, and I will explore it in another post.

Recommended Reading

I have just finished reading Cal Newport’s excellent books “Deep Work” and “Digital Minimalism“, and while both are somewhat related to what has been discussed here, in particular the latter book provides simple practical strategies to overcome the social media addiction. Even though I have been already “very minimal” on social networks, I have just learned some crucial patterns that I was missing before. For example, I’m going to follow Cal’s recommendation to completely stop upvoting/downvoting – I can see now that it has been causing some anxiety for me, as I had to find an internal justification for downvoting, which didn’t feel good at times and felt like I was engaging in cowardly behavior. And also feeling bad when someone downvoted my contribution. Completely eradicating the participation in up/downvoting process removes all anxiety and saves a ton of time. I still know what I like and what I dislike, but I don’t have to make an anonymous statement about it. Yay! Thank you, Cal!

These days, with extremely powerful realistic text-generating machine learning engines like GPT-2, this technique becomes more and more automated and no longer requires networks of humans to create fake things. It’s the domain of AI agents these days.

But, in the domain of online stores these machinations are still tricky to do, since unlike online profiles that have no physical boundaries, real products are bound by hard coins and thus it’s much more difficult to game. But, of course, it’s being gamed.

The name of the game is generating multiple positive reviews by buying them with freebies.

While in the past a merchant could pay anybody to go and write reviews on sites like Amazon and all was good, recently machine learning started to being deployed to ignore those reviews, and instead count only reviews with the ‘Verified Purchase’ tag. i.e. only reviewers that spend their hard-earned cash are considered as real. Surely, the ML model also validates that the verified purchaser has a history of normal purchases and wasn’t created just to game the system, even if the money was spent for real – (create a new account, send to it some credit via a gift card, buy the product, forget about the account).

Having recently purchased a few hardware products on Amazon, I discovered that the Chinese merchants found a way to beat the system and bypass the AI gatekeeper. Once you buy say some grow lights, when your product arrives it will include a variety of incentives, such as:

1. write a 5-star review and receive a free gift
2. join the testers and receive free products in exchange for reviews

Who doesn’t want a free gift? So a lot of those 5-star reviews were “bought” with a free gift bribe. Notice, that Amazon ships those products and it knows that the incentive card is there. Don’t you find it strange?

Then, there is the free-testers program, where you get freebies in exchange for reviews. In order to fly below the fake review detector, the customer is told to purchase the product as if they wanted it in first place, then post a review, then email the company with the review and the receipt and they refund you for the full cost via paypal. Now, again, when you get the whole product for free, will you leave a bad review? So here, again, we get a bunch of fake reviews. Do you think Amazon doesn’t know about those?

I’m probably being unfair here instigating that it’s only the Chinese companies that do that. To clarify, I’m only sharing my direct experience and it came through Chinese companies so far.

I had another incident on Amazon, where I purchased a really badly engineered product and it fell apart after a few uses, so I left a negative review, listing its shortcomings. Next, the vendor of that product started hassling me over email trying to bribe me to change my rating from 2 to 4. This makes no sense to me. If the vendor were to fix the problems in the product and to send me a better product, I would have been more than happy to give them a glowing review, but paying me off to say lies is just lame. Though, I’m sure that it did work on some other reviewers.

Bottom line, a lot of reviews on Amazon and other sites are fake and they are there to stay. Let’s see how Amazon’s attempts at using machine learning to discount fake works out, this is a very exciting time. And, surely, the merchants will have to find adversarial ways to beat the machine.

Gaming reviews has been a big issue for any marketplace, and in particular on Amazon. Amazon does a pretty so-so job at weeding out fake reviews out as it is gamed freely by some Chinese vendors in an inventive way that appears legit. But it has recently deployed a new strategy of how it weighs each given review. Reviews with no ‘Verified Purchase’ tag get a lesser weight than those with the tag, and older reviews get progressively less weight in the game.

(note: I have no direct source to confirm that this is so, and I derived this by looking at multiple listing, asking for help from others and doing the math, which after several days of investigation lead to what appears to be a good understanding of what is probably going on. So please don’t take this as a definitive answer.)

Let’s Destroy my Book’s Reputation

My first book was published in 2003. Now long forgotten by most, the Apache/mod_perl technology became outdated around 2010, and moreover, this particular book was written about the first generation of this technology, and which got superseded by the second generation in 2004. As you can tell this book is not relevant for today’s tech world, unless you happen to maintain a 15-year old system that refuses to quit.

Until recently, that book had an almost excellent total 4.4 rating with 3 5-star reviews, and 4 4-star reviews. Which I think reflects well the reality. It’s an excellent work of 900 pages that took me 3.5 years of hard work to create. Once I finished the writing, I discovered that writing technology books is a foolishness, because they get outdated before they hit the presses. It’s better now, when books are digital, as compared to dead trees. But even with digital books, for an author to constantly update the book to match the ever-changing at a furious speed APIs and validating that the code still works is an insanely time-wasteful effort.

Now, I know all those reviews on my book were honest and I have never solicited those. Only one of those reviews has a Verified Purchase tag, but many of my books were purchased at the O’Reilly Open Source Convention – I know that, because for 7 years I taught workshops at that conference and signed many of those books that were purchased there. Powell Books was usually the book vendor there.

Someone recently rated my book at 1-star (no review). I can’t imagine, who in their right mind would buy a 17-year old book on a long dead technology, and expect something good out of it? No wonder they must have been enraged when they started reading it and as a result left a I-hate-it 1-star rating. It could have been done out of pure malice too. Who knows.

You might say, what’s the big deal. Indeed, it shouldn’t be a big deal, since no single review should define a book’s quality – but, unfortunately, it is a big deal. Since after that rating was added, the total rating of the book went from 4.4 to 2.9.

Now, you might say, no way. But, unfortunately, numbers don’t lie:

So we have 7 positive and 1 negative ratings and you can see the system gave a whooping 48% weight to a single negative rating! Don’t you find it just odd?

Of course, the book has no chance to balance this machinated wrong-doing, because other than a straightforward manipulation of me asking someone to buy the book, rating it at 5-stars and refunding them the money, nobody in their sane mind is going to do it on their own will, because the book is 17 years old and is no longer relevant.

You might say that, isn’t this a good thing that no longer relevant books get bad rating? It’s a difficult question. It depends on who is looking:

• To a potential buyer of this specific old book it’s a good rating – signaling don’t buy.
• To an author who worked hard to create the book this is a really bad rating as it doesn’t reflect the quality of their work.
• To someone who might be looking at author’s page before buying his/her other books this is a bad rating, since who wants to take the risque of buying new books if their other books have a bad rating, and this is clearly a misrepresentation of the truth.
• To a potential publisher who might be considering giving this author a contract for a new book this is a bad rating.

Moreover, depending on whether you’re logged in or not, Amazon is likely to show you a different rating. For example, my book shows 3.1/5 when I’m logged out.

Note, how the 1-star rating went from 48% in weight to 41%. Why do I get a different rating whether I’m logged in or not?

International Reviews

Another thing you may notice is that Amazon started incorporating International reviews and ratings into the total ratings. This too is very confusing for some books, since you may get a totally different rating depending on whether you’re logged in or not. For example, this book shows a rating of 5/5 if you aren’t logged in (and sometimes even if you are logged in):

But when I log in and I go to the book’s page, it drops from 5/5 to 3/5:

Here, despite 9 5-star reviews, the book is ranked at 3/5. When I log in, a single 2-star International review is being displayed, with a crazy disproportionate weight of 68%, which brings a 9 5-star reviews book, to 3.0. Can you make sense of it? Do you read the fantastic reviews and buy the book, or do you run like hell because Amazon thinks it’s a terrible book – based on a single recent Verified purchase international review?

Have I mentioned that none of those 9 5-star reviews are ‘Verified Purchase’? Bad signal for sure – we don’t trust reviews from those who didn’t pay us money, Amazon communicates.

Note, that this example is not of a technical book, so something else must be going on.

Customer Service Blames Everything on Machine Learning

I contacted Amazon customer service, asking for help to understand what’s going on and got a reply:

The overall star rating for a product is determined by a machine-learned model that considers factors such as the age of the review, helpful votes by customers, and whether the reviews are from verified purchasers. Similar machine-learned factors help determine a review’s ranking in the list of reviews. The system continues to learn which reviews are most helpful to customers and improves the experience over time. Any changes that customers may currently experience in the review ranking or star ratings is expected as we continue to fine-tune our algorithms.

This doesn’t help me, as a customer to understand what’s going on.

Please Remove my Books from Amazon

One thing for sure, as an author, I don’t want my book to be misrepresented and I’m going to ask O’Reilly Media, Inc, the publisher, to pull my book off Amazon and I’m going to try to find a way for Amazon to remove all books published by me and never sell any of my books on Amazon. Amazon banned me from self-publishing a few years ago, because of some internal conflict with a 3rd party vendor, which had nothing to do with me, and without giving me any reason of why suddenly I wasn’t a kosher writer, while continuing selling my other books. Now this situation should hopefully remove any income coming from my works to Amazon. Surely, I will sell less books without Amazon, but I prefer that, to Amazon transmitting falsified information about my works.

Recommendations for Tech Book Authors on Amazon

To conclude, here is what you need to know as an author to prevent Amazon from burying your works over time:

• Your older reviews get less and less valid with time – you need to find a way to get recent reviews (hint, hint)
• When you (hint, hint) get them, make sure they are posted by those who purchased the book on Amazon. Should they buy it elsewhere, too bad – they will have to buy another copy on Amazon. Basically, Amazon forces you to game its system, while making money for Amazon.
• If your book is aging, which in particular impacts technology authors, but I’m sure it affects other categories as well, its reputation is going to get destroyed by recent negative reviews, which you can’t avoid. Because the relevancy of your work is diminishing, the 1-star reviews will prevail. The only recourse here at the moment to keep your reputation unblemished, I think, is to find a way to remove your book from Amazon. I haven’t investigated other marketplaces to tell whether it’s the same situation over there, but I’m sure all markets will follow Amazon’s lead. So, it’s best to remove the aged book altogether and not allow automated systems to misrepresent the child of your hard work.

Recommendations for Marketplace Designers

I doubt Amazon would care for my recommendations, but you never know, someone on their dev team might read them and at least consider them.

After contemplating the current situation here are some recommendations that I came up with:

• When you introduce weights for different signals, visible to users, make sure the outcome makes sense. e.g. the second book presented in this article has a rating of 3.0 and it displays 9 5-star reviews and nothing else – weird, no?
• Weigh ratings w/o reviews at a lower weight than ratings with reviews. It’s too easy to submit a number on the rating scale without giving it much thought, usually resulting in less than reality-reflecting ratings (in both good and bad directions). It takes time and thinking to write a review, so usually ratings bundled with reviews are of a higher fidelity.
• Weigh higher reviews with real customer names as compared to Anonymous reviews. It’s easy to write a negative review when one intends malice (competitor/hater/etc.). If you read listings you’d find few positive reviews by Anonymous. Surely, some people don’t feel safe to write a totally valid negative review with their name on it for fear of persecution. So this recommendation is highly debatable.
• Apply all the weighing smarts for products with dozens and hundreds of reviews, the truth may prevail there (unless it was gamed), but be careful with niche products/books with under 10-15 reviews. Any automated weight re-balancing of products with very few reviews can lead to a disastrous outcome for the creator. A carefully thought out different algorithm or a ML model should be applied there.
• Give authors of technology books (and whatever similar domains fit here) a chance to indicate when a book is outdated, to prevent disappointments from users and almost guaranteed ensuing negative reviews/ratings. The book should probably still be available for some years to support those who maintain old systems. But it should be made loud and clear to the customers, that they are buying an outdated manual before they hit the ‘Pay’ button. That is if the author was given a way to signal this situation.
• Separate second hand market ratings from the normal books. Very often I see a 1-star post from a second-hand book buyer: “I received a damaged copy”. What does this rating have to do with the quality of the contents of the book. This is very damaging to the total book rating. Why do you do nothing about this? Perhaps, the rating form can be adjusted to indicate whether the user provides feedback on the physical quality of the book as compared to the content.

I honestly don’t know what to do about the ‘Verified purchase’ reviews outweighing the reviews w/o that tag. From one point, it’s easier to add fake reviews w/o purchasing the product. On the other hand it’s just as easy to make people buy a book/product, write a review and then refund them the cost via paypal (a common current way to game the system). And as shared earlier in this article, this forces more sales for Amazon, and discourages competition. I trust Amazon has a good insight on this one.

This situation took several days to sort out and I’m glad the picture is finally clear.

The automated future is looking is great for the masses. Niche creators watch out!

Today one of the scammers almost got me.

I do healing work amongst other things and today I received what seems to be a normal contact from a potential client with a slightly odd side to it.

Hello, i’m kylie , i would like to know if you are a Wellness Coach or Reiki Healing and if you accept credit card for payment?

I follow up, yes, sure. I get next:

the treatment is for 4 ladies in my family, pls get back to me with the total estimate for 2 times in a week 4 (Females) for 4 weeks and i want the session to be 1 hour, and i will like to know the total cost for the service

would prefer you treat them on

Depression

weight loss

reducing pain and discomfort?

nutrition

stress management and Anger management

relationships

both and I want you to know that I am diagnosed of lung cancer and i’m currently in the hospital right now due to my health issue and i want to secure this appointment for my family as a surprise to console them and keep them busy while am undergoing my surgery so i hope you can handle them very well? … could kindly get back to me with the total cost for 4 ladies times.2 sessions per week, times 4 weeks which is 32 sessions

am okay with the price ,i will be making the payment now to secure the appointment and please i will like you to help me with little favor ?

here is the favor i need from you…
I want you to charge my credit card for the sum of $4750,once the money clear you are to keep $2,100 as payment for the services and help me send $2500 to my private Limo driver, via cash deposit and the money is for refreshment,transportation and accommodation fees he will also be their guardian for the entire weeks of the coaching
and am making this as a huge surprise to console them for my upcoming surgery operation, and i will like you to hold the remaining $150 as a tips for handling this for me,Thanks

As of this writing I have published 4 books. First with O’Reilly and Associates, second with Onyx Neon Press, and the the most recent two I self-published via Amazon KDP (Kindle Digital Publishing).

Publishing via KDP was OK, they still have a long way to go to make things smooth, the main gift to us was to make it possible to publish books without needing to convince a mainstream publisher to want to publish them. When things work, they work, when there are problems, usually after stumbling through a few clueless support people, eventually you hit someone who knows their business and things get resolved. All was good.

Recently I started studying a fascinating science of Reichian Therapy, and stumbled upon a manuscript by Jack Willis, called “Reichian Therapy – The Technique, For Home Use”.

He released his manuscript for free as a pdf on his website in 2008. The manuscript allows anybody to republish it for free or a for profit. I quote from page 2 of the pdf:

Copyright (C) 2007 by Jack Willis.
Edition 3 posted June 2008
Authorization to reprint: This work may be copied, distrib-
uted, published and republished by any individual or entity for
free or as a commercial venture without payment of royalty fees
to the author. The book may be reformatted as need for publica-
tion. Pictures may be substituted with identical pictures using
different models. The only copyright restriction is that the text
may not be changed for copied, distributed, published or repub-
lished copies or editions. If reformatted for publication, the
table of contents and the index may be re-calculated or omitted
at the discretion of the person or entity doing the reformatting
for publication.

Jack Willis passed away in 2010 and his site is now gone, but luckily waybackmachine did it again and it has a copy of the site here: http://reichiantherapy.net.

I wanted to make the text readable on an ebook reader, so I OCRed the pdf, fixed spelling errors and typography issues, and spent a lot of time reformatting it for a digital book usage, merging paragraphs, moving images and footnotes to where they belonged in the text, etc.

After that I decided that others will probably benefit from this text as well in its digital format and decided to upload it to my KDP bookshelf. This was something new, and I read through the materials KDP supplied on the issue of copyright, it seemed to me that based on the quoted above authorization from the author I was doing the right thing, so I made a nice book cover for it and uploaded the manuscript.

At the moment of my uploading there was only a print version of the book available from a large publisher “New Falcon Publications”, which they published in August 21, 2013, and which they sell for $40. They renamed the book to: “Reichian Therapy: A Practical Guide for Home Use“.

After several days of waiting I received an email from KDP support:

Hello,

Thank you for publishing with Amazon. Copyright is important to us – we want to make sure that no author or other copyright holder has his or her books sold by anyone else. To publish your book, please respond with documentation confirming your publishing rights within four days:

Reichian Therapy: The Technique, for Home Use by Willis, Jack (AUTHOR) (ID:9084681)

Acceptable documentation includes:

– A contract or statement from the author or publisher verifying you retain publishing rights
– An e-mail from the address listed on the official author or agent’s website
– For authors using a pseudonym, copyright registration or statement of pseudonym use

If you publish books for which you do not hold the publishing rights, your account may be terminated.

I supplied the documentation emailing them the pdf released by the author and quoting the authorization I pasted earlier in this article.

A day later I received another email from KDP support:

Hello,

Thank you for the information you provided regarding the following book(s):

Reichian Therapy: The Technique, for Home Use (ID:9084681)

Prior to your submission, we received a notice and takedown for a book that matches to yours, from a third party claiming that the distribution of the book above was not properly authorized due to copyright infringement.

We don’t involve ourselves in third party disputes and because we have not received any communication from the involved parties that the matter has been resolved, we have determined that we will not be making the book(s) available for sale on Amazon at this time.

We appreciate your understanding in this matter.

As a result of that, the entry for that book on my Amazon KDP bookshelf had a big BLOCKED on it (unfortunately no snapshot).

After a bit of thinking I realized that probably the conflict they referred to was with “New Falcon Publications”, who obviously doesn’t want competition, yet they won’t make the effort to release a kindle version of the book. Why? Let me guess – probably because $40 is much better than maximum $9.99 one can sell for a kindle version of any book. My only guess is that whoever published the digital version before me received a takedown notice from “New Falcon Publications”, because if you follow the money trail, there is nobody else who currently profits from this book.

I replied to KDP support, saying that I understand the situation and please remove that blocked entry from my bookshelf.

The same day I received a reply:

Hello,

At this time, it is not possible to completely remove an unpublished book from the Bookshelf. I’m sorry if this may cause you any inconvenience.

OK, I said, weird, but what to do. I moved on.

Having done the work, I didn’t want it to go wasted, so I made it available for free on my website under Reichian Therapy.

Two weeks passed and I get a new email from KDP:

Subject: Your Amazon KDP Account

Hello,

Due to copyright infringement and Content Guideline violations with respect to books you have submitted through your account, specifically, the submission of content for which you did not have the necessary rights, we are terminating your account and your Agreement effective immediately. Here are some examples of the book(s) that you submitted through your account that fall into this category:

Reichian Therapy: The Technique, for Home Use, by Willis, Jack (Title ID: 9084681)

As part of the termination process, we will close your KDP account and the related CreateSpace account (if any) and remove the books you have uploaded through our channels from sale on Amazon. Note that you are no longer eligible to receive unpaid royalties for sales that occurred prior to this termination.

Additionally, as per our Terms and Conditions, you are not permitted to open any new KDP and/or CreateSpace accounts.

If you have any questions, please email us at content-review@amazon.com.

Best Regards,

Claudia Z.

Say what?! That was odd, so I sent a reply asking for a clarification, since it was very odd that they decided that I infringed on a copyright when I sent them the manuscript’s authorization that allows anybody to republish it. A few days later I received no clarification, but a different version of the same:

Subject: Your Amazon KDP Account

Hello,

Thank you for the email concerning the status of your account.

After reviewing your response, we have reevaluated the Content Guideline violations relating to the books in your account.

We found that you have uploaded material through your account for which you do not have the necessary rights.

As a result, we are upholding our previous decision to terminate your KDP account and remove all your books from Amazon.

If you have any questions regarding this issue, please email us at content-review@amazon.com.

If you would like to review our Content Guidelines, please visit: https://kdp.amazon.com/self-publishing/help?topicId=A2TOZW0SV7IR1U

Best Regards,

Luca F

I hope you can see the obvious.

I next called KDP trying to talk to someone, who could explain, what the real problem was. You can’t call KDP directly, so you call normal Amazon customer support 1-866-216-1072 and press 0 to talk to an agent. This is general customer service, tell them you want to talk to KDP, they will check your identity and transfer you to KDP. Ideally, have some recent amazon.com order number available if you have such, otherwise it’s going to take one hour or so for the usually incompetent agent somewhere in 3rd world to find and validate your account. One time the agent claimed that I was shopping with Amazon Mexico! When I’m from Canada and have never used Amazon Mexico in my life.

After the very inefficient process of validation I was redirected to someone who listened to my story and told me that that my account got closed due to a ransomware virus on my computer. He was very sure of that. It took me a few moments of bewildered surprise to connect the dots and realize I was talking to a wrong person. After realizing that, I discovered I wasn’t redirected to KDP support, but I was talking to yet another clueless person at tech support.

An hour later with a bunch more validations I reached someone at KDP. It immediately felt that I reached a higher place, since the tone of voice on the other side was intelligent and caring for a change. I was relieved. After telling my situation I was redirected to yet another person, who listened to me, said that there is no way to contact the content review team and that he (Chris) will inquire for me and get back to me in 2 business days. I said thank you very much and waited.

One odd thing Chris told me that if the content review team doesn’t solve the problem he said just open another account and we will transfer all the book reviews from the old account to the new one. This is despite content review team threatening me in every email from them that I’m not allowed to open another account with KDP and if I do they will never pay me a cent. Hello, Amazon?! Can you decide on your policies and not provide misleading information?

A week passed by, and nobody contacted me. So I contacted KDP again, again going through many agents before reaching the right people, patiently waiting for the strangely untrained agents who again and again asked me to validate my address and often not being able to do so. Yet, eventually I reached KDP again and telling my story twice I was redirected to someone senior (Karthis), who again asked me to tell the story, and said it makes sense and that again he can’t talk to anybody at content review and that he will prioritize this issue for them to get back to me within 2 biz days.

Another week passed by with no reply from anybody. I called KDP again, [fast forward the painful reaching them process], and after several bounces again reached Karthis, who promised the same thing. I objected and said that nothing will be different, and can you please talk to someone in person and tell me the real reason for the termination. The answer was the same, he will prioritize it even more.

Luckily this time I did receive a reply the next day, which gave me no new information, other than repeating its old threats:

Hello,

We’ve reviewed the information you provided and we are upholding our previous decision to terminate your account and remove all your books from sale on Amazon.

As a result, we will not be reinstating your account.

Please note that, per our Terms and Conditions, you are not permitted to open new accounts and will not receive future royalty payments from additional accounts created.

Best Regards,

John M

Obviously nobody wants to tell me the real reason for the false allegation.

Amazon, wake up! We authors make money for you! You won’t have existed if it weren’t for people giving you stuff to sell. We appreciate the platform and the convenience, but you have just shut down an author, who wrote honest books, for some kind of mysterious conspiracy BS, that you don’t even have the guts to disclose.

Also notice despite the “we are upholding our previous decision to terminate your account and remove all your books from sale on Amazon” that Amazon did not remove all my books from sale on Amazon.com, they only removed the books that I self-published. The two other books that I have authored, Amazon continues selling because they have been published by big publishers:

1. Practical mod_perl
2. mod_perl 2 User’s Guide

So Amazon still sells books by a criminal, which they alleged me to be. This is so inconsistent and illogical.

The bottom line is that my effort to make a copyright-free digital work, that is no longer available, accessible to public, backfired with Amazon shutting down my account and banning me from ever publishing books with it. Very nice job, Amazon.

I don’t really care for my account reinstating, since I will not ever work with Amazon KDP again out of my own choice. I don’t work with people or companies, who lack integrity, play ambiguous games and display so much incompetence. It’s just sad that the company won’t honestly communicate with authors, who they rely on to make their profit.

Good bye, Amazon KDP.

p.s. I patiently waited for almost 3 weeks hoping this gets resolved, but clearly the one thing KDP wouldn’t do is to admit its mistake and reinstate my account. Meanwhile my books have been unavailable for sale. I need to start looking for a new publisher. If you have a good experience with a different publisher, please kindly share. My recent 2 books are in the self-help/health category.

Higbie was the first person to profit by my great and infallible scheme for finding work for the unemployed. I have tried that scheme, now and then, for forty-four years. So far as I am aware it has always succeeded, and it is one of my high prides that I invented it, and that in basing it upon what I conceived to be a fact of human nature I estimated that fact of human nature accurately.

Higbie and I were living in a cotton-domestic lean-to at the base of a mountain. It was very cramped quarters, with barely room for us and the stove-wretched quarters indeed, for every now and then, between eight in the morning and eight in the evening, the thermometer would make an excursion of fifty degrees. We had a silver-mining claim under the edge of a hill half a mile away, in partnership with Bob Howland and Horatio Phillips, and we used to go there every morning carrying with us our luncheon, and remain all day picking and blasting in our shaft, hoping, despairing, hoping again, and gradually but surely running out of funds. At last, when we were clear out and still had struck nothing, we saw that we must find some other way of earning a living. I secured a place in a near-by quartz mill to screen sand with a long-handled shovel. I hate a long-handled shovel. I never could learn to swing it properly. As often as any other way the sand didn’t reach the screen at all, but went over my head and down my back, inside of my clothes. It was the most detestable work I have ever engaged in, but it paid ten dollars a week and board-and the board was worth while, because it consisted not only of bacon, beans, coffee, bread and molasses, but we had stewed dried apples every day in the week just the same as if it were Sunday. But this palatial life, this gross and luxurious life, had to come to an end, and there were two sufficient reasons for it. On my side, I could not endure the heavy labor; and on the Company’s side, they did not feel justified in paying me to shovel sand down my back; so I was discharged just at the moment that I was going to resign.

If Higbie had taken that job all would have been well and everybody satisfied, for his great frame would have been competent. He was muscled like a giant. He could handle a long-handled shovel like an emperor, and he could work patiently and contentedly twelve hours on a stretch without ever hastening his pulse or his breath. Meantime, he had found nothing to do, and was somewhat discouraged. He said, with an outburst of pathetic longing, “If I could only get a job at the Pioneer!”

I said “What kind of a job do you want at the Pioneer?”

He said “Why, laborer. They get five dollars a day.”

I said “If that’s all you want I can arrange it for you.”

Higbie was astonished. He said “Do you mean to say that you know the foreman there and could get me a job and yet have never said anything about it?”

“No” I said, “I don’t know the foreman.”

“Well” he said, “who is it you know? How is it you can get me the job?”

“Why,” I said, “that’s perfectly simple. If you will do as I tell you to do, and don’t try to improve on my instructions, you shall have the job before night.”

He said eagerly “I’ll obey the instructions, I don’t care what they are.”

“Well,” I said, “go there and say that you want work as a laborer; that you are tired of being idle; that you are not used to being idle, and can’t stand it; that you just merely want the refreshment of work, and require nothing in return.”

He said “Nothing?”

I said, “That’s it-nothing.”

“No wages at all?”

“No, no wages at all.”

“Not even board?”

“No, not even board. You are to work for nothing. Make them understand that-that you are perfectly willing to work for nothing. When they look at that figure of yours that foreman will understand that he has drawn a prize. You’ll get the job.”

Higbie said indignantly, “Yes, a hell of a job.”

I said, “You said you were going to do it, and now you are already criticising. You have said you would obey my instructions. You are always as good as your word. Clear out, now, and get the job.”

He said he would.

I was pretty anxious to know what was going to happen-more anxious than I would have wanted him to find out. I preferred to seem entirely confident of the strength of my scheme, and I made good show of that confidence. But really I was very anxious. Yet I believed that I knew enough of human nature to know that a man like Higbie would not be flung out of that place without reflection when he was offering those muscles of his for nothing. The hours dragged along and he didn’t return. I began to feel better and better. I began to accumulate confidence. At sundown he did at last arrive and I had the joy of knowing that my invention had been a fine inspiration and was successful.

He said the foreman was so astonished at first that he didn’t know how to take hold of the proposition, but that he soon recovered and was evidently very glad that he was able to accommodate Higbie and furnish him the refreshment he was pining for.

Higbie said “How long is this to go on?”

I said “The terms are that you are to stay right there; do your work just as if you were getting the going wages for it. You are never to make any complaint; you are never to indicate that you would like to have wages or board. This will go on one, two, three, four, five, six days, according to the make of that foreman. Some foremen would break down under the strain in a couple of days. There are others who would last a week. It would be difficult to find one who could stand out a whole fortnight without getting ashamed of himself and offering you wages. Now let’s suppose that this is a fortnight-foreman. In that case you will not be there a fortnight. Because the men will spread it around that the very ablest laborer in this camp is so fond of work that he is willing and glad to do it without pay. You will be regarded as the latest curiosity. Men will come from the other mills to have a look at you. You could charge admission and get it, but you mustn’t do that. Stick to your colors. When the foremen of the other mills cast their eyes upon this bulk of yours and perceive that you are worth two ordinary men they’ll offer you half a man’s wages. You are not to accept until you report to your foreman. Give him an opportunity to offer you the same. If he doesn’t do it then you are free to take up with that other man’s offer. Higbie, you’ll be foreman of a mine or a mill inside of three weeks, and at the best wages going.”

It turned out just so-and after that I led an easy life, with nothing to do, for it did not occur to me to take my own medicine. I didn’t want a job as long as Higbie had one. One was enough for so small a family-and so during many succeeding weeks I was a gentleman of leisure, with books and newspapers to read and stewed dried apples every day for dinner the same as Sunday, and I wanted no better career than this in this life. Higbie supported me handsomely, never once complained of it, never once suggested that I go out and try for a job at no wages and keep myself.

That would be in 1862. 1862 I parted from Higbie about the end of 62 or possibly it could have been the beginning of 63 and went to Virginia City, for I had been invited to come there and take William H. Wright’s place as sole reporter on the Territorial Enterprise and do Wright’s work for three months while he crossed the plains to Iowa to visit his family. However I have told all about this in “Roughing It.”

I have never seen Higbie since, in all these forty-four years.

Shortly after my marriage, in 1870, 1870 I received a letter from a young man in St. Louis who was possibly a distant relative of mine-I don’t remember now about that-but his letter said that he was anxious and ambitious to become a journalist-and would I send him a letter of introduction to some St. Louis newspaper and make an effort to get him a place as a reporter-It was the first time I had had an opportunity to make a new trial of my great scheme. I wrote him and said I would get him a place on any newspaper in St. Louis; he could choose the one he preferred, but he must promise me to faithfully follow out the instructions which I should give him. He replied that he would follow out those instructions to the letter and with enthusiasm. His letter was overflowing with gratitude-premature gratitude. He asked for the instructions. I sent them. I said he must not use a letter of introduction from me or from any one else. He must go to the newspaper of his choice and say that he was idle, and weary of being idle, and wanted work-that he was pining for work, longing for work-that he didn’t care for wages, didn’t want wages, but would support himself-he wanted work, nothing but work, and not work of a particular kind, but any kind of work they would give him to do. He would sweep out the editorial rooms; he would keep the ink-stands full, and the mucilage bottles, he would run errands, he would make himself useful in every way he could.

I suspected that my scheme would not work with everybody-that some people would scorn to labor for nothing, and would think it matter for self-contempt; also that many persons would think me a fool to suggest such a project; also that many persons would not have character enough to go into the scheme in a determined way and test it. I was interested to know what kind of a candidate this one was, but of course I had to wait some time to find out. I had told him he must never ask for wages; he must never be beguiled into making that mistake; that sooner or later an offer of wages would come from somewhere, and in that case he must go straight to his employer and give him the opportunity to offer him the like wages, in which case he must stay where he was-that as long as he was in anybody’s employ he must never ask for an advance of wages; that would always come from somewhere else if he proved his worthiness.

The scheme worked again. That young fellow chose his paper, and during the first few days he did the sweeping out and other humble work; and kept his mouth shut. After that the staff began to take notice of him. They saw that they could employ him in lots of ways that saved time and effort for them at no expense. They found that he was alert and willing. They began presently to widen his usefulness. Then he ventured to risk another detail of my instructions; I had told him not to be in a hurry about it, but to make his popularity secure first. He took up that detail now. When he was on his road between office and home, and when he was out on errands, he kept his eyes open and whenever he saw anything that could be useful in the local columns he wrote it out, then went over it and abolished adjectives, went over it again and extinguished other surplusages, and finally when he got it boiled down to the plain facts with the ruffles and other embroideries all gone, he laid it on the city editor’s desk. He scored several successes, and saw his stuff go into the paper unpruned. Presently the city editor when short of help sent him out on an assignment. He did his best with it, and with good results. This happened with more and more frequency. It brought him into contact with all the reporters of all the newspapers. He made friends with them and presently one of them told him of a berth that was vacant, and that he could get it and the wages too. He said he must see his own employers first about it. In strict accordance with my instructions he carried the offer to his own employers, and the thing happened which was to be expected. They said they could pay that wage as well as any other newspaper-stay where he was.

This young man wrote me two or three times a year and he always had something freshly encouraging to report about my scheme. Now and then he would be offered a raise by another newspaper. He carried the news to his own paper; his own paper stood the raise every time and he remained there. Finally he got an offer which his employers could not meet and then they parted. This offer was a salary of three thousand a year, to be managing editor on a daily in a Southern city of considerable importance, and it was a large wage for that day and region. He held that post three years. After that I never heard of him any more.

About 1886 my nephew, Samuel E. Moffett, a youth in the twenties, lost his inherited property and found himself obliged to hunt for something to do by way of making a living. He was an extraordinary young fellow in several ways. A nervous malady had early unfitted him for attending school in any regular way, and he had come up without a school education-but this was no great harm for him, for he had a prodigious memory and a powerful thirst for knowledge. At twelve years he had picked up, through reading and listening, a large and varied treasury of knowledge, and I remember one exhibition of it which was very offensive to me. He was visiting in our house and I was trying to build a game out of historical facts drawn from all the ages. I had put in a good deal of labor on this game, and it was hard labor, for the facts were not in my head. I had to dig them painfully out of the books. The boy looked over my work, found that my facts were not accurate and the game, as it stood, not usable. Then he sat down and built the whole game out of his memory. To me it was a wonderful performance, and I was deeply offended.

As I have said, he wrote me from San Francisco in his early twenties, and said he wanted to become a journalist, and would I send him some letters of introduction to the newspaper editors of that city-I wrote back and put him strictly under those same old instructions. I sent him no letter of introduction and forbade him to use one furnished by anybody else. He followed the instructions strictly. He went to work in the Examiner, a property of William R. Hearst. He cleaned out the editorial rooms and carried on the customary drudgeries required by my scheme. In a little while he was on the editorial staff at a good salary. After two or three years the salary was raised to a very good figure indeed. After another year or two he handed in his resignation-for in the meantime he had married and was living in Oakland, or one of those suburbs, and did not like the travel to and fro between the newspaper and his home in the late hours of the night and the morning. Then he was told to stay in Oakland, write his editorials there and send them over, and the large salary was continued. By and by he was brought to New York to serve on Mr. Hearst’s New York paper, and when he finally resigned from that employment he had been in Mr. Hearst’s employ sixteen years without a break. Then he became an editorial writer on the New York World with the privilege of living out of town and sending his matter in. His wage was eight thousand dollars a year. A couple of years ago Collier’s Weekly offered him an easy berth and one which was particularly desirable in his case, since it would deal mainly with historical matters, past and present and that was an industry which he liked. The salary was to be ten thousand dollars. He came to me for advice, and I told him to accept, which he did. When Mr. Pulitzer found that he was gone from the World he was not pleased with his managing editor for letting him go, but his managing editor was not to blame. He didn’t know that Moffett was going until he received his resignation. Pulitzer offered Moffett a billet for twenty years, this term to be secured in such a way that it could not be endangered by Pulitzer’s death, and to this offer was added the extraordinary proposition that Moffett could name his own salary. But of course Moffett remains with Collier, his agreement with Collier’s having been already arrived at satisfactorily to both parties.

Written by Mark Twain on Tuesday, March 27, 1906 in Autobiography of Mark Twain Vol 1. pages 447-451.

The amazing Autobiography of Mark Twain was published 100 years after his death. Samuel Langhorne Clemens (aka Mark Twain) willed that – so that he could write freely and without censorship to make sure his words won’t “hurt” anybody alive with his honest and direct account of the matters at hand. This unusual-unconstrained style of delivery for that era makes it a very delightful read. The full text is also available freely at marktwainproject.org.

If you have used this strategy or know someone who did please share with the rest of the readers. Thank you.

Running a small business can be tough, especially in the current economy conditions where clients aren’t easy to come by, so keeping your expenses to the minimum is a certain way to improve your bottom line. When it comes to promotional materials, one could spend a lot of money there. And business cards are usually the most commonly used way to promote one’s business. It’s the magic handshake protocol most entrepreneurs are so familiar with:

– “What do you do?”

– “I’m a Registered Massage Therapist. What about you?”

– “I’m a Reiki Master, and here is my business card. Can I have your card?”

– “Of course, here it is.”

The Internet has been around for a relatively long time by now, yet, business cards aren’t a thing of a past and it’ll probably take quite some time before one will not carry any around with them anymore and not leave a few here and there on various posting boards in cafés and community places.

Business cards have been around for centuries (In China since 15th century, Europe – 17th century) . So it’s not surprising that these days there are many shops that will print your business cards in various shapes and forms, using different fancy kinds of paper and color. But they can be quite expensive, especially if someone designs them from scratch – since you’re paying for their time, their rent costs, etc.

I’d like to introduce you to a different way of getting your cards done and for free (only paying for the shipment). I have been using VistaPrint for probably at least 5 years now. I’d get their 250 free business cards package about once a year and each time only pay for the shipment. You design your own card using their online card design software, which is not fancy but it does what you need – it lets you put the essential information on the business card, while making it look good. Here is one of the recent cards I’ve designed using vistaprint service.

They have a limited choice of backgrounds which are included in the free business card package and for a small extra cost you can add your own or browse through hundreds of other background designs. It’s tempting to spend a little extra, but for me I always was able to just choose one I liked from the free ones and go with it.

Once you’re happy with your card’s design, starts the process of going to the check out page. If you’ve ever been to Ikea, where they drag you through the whole shop to show you all of their wares, the process is similar here. They will try to upsell you 10 other things before you reach to the page where you do your payment: magnets, mugs, stickers, etc., etc.. Unless you really need any of those things, simply quickly press “Next” and get to the checkout page, pay and be done with it. The only cost here is the shipment which usually comes to about $9. It looks like they ship pretty much anywhere in the world.

The only one important thing to mention is that they will offer you to have a blank back side of the card for something like $5 extra, as compared to the little one-line text advertising of vistaprint site on the back of the card. I won’t pay the $5, because I actually like having that advertising there and I tell anybody I meet about the great deal vistaprint gives to people and show them the back of the card so they won’t forget the url of the site. I think vistaprint should pay me to have that ad on the back on the card, rather than me paying them to remove it, since I actively help them to promote their business. If you really don’t want the little ad, I’d say choose to put the calendar for the coming year, people are more likely to carry your card around, as small year calendars are always handy, even in the digital edge.

Once you paid – the package should arrive within a few weeks. I ordered probably about 10 times from them so far and never had any quality or delivery problems.

As I have only ever used one free business card printing company I don’t know of any others I can recommend, so if you have other recommendations for print shops that give away free business cards, let me know and I’ll add the link to other companies as well. Thank you.