Royce Bits

The Nested Mechanical Turk

noreply@blogger.com (Royce Williams (TychoTithonus)) — Sat, 19 Apr 2025 22:48:00 +0000

(Speculation, and not the whole story ... but fits some facts):

For more than a decade, foreign influence operations have used techniques from technology, marketing, and social media -- tracking interaction/propagation, correlating with marketing metadata, A/B testing, etc. -- to map people to their "trigger" topics: ones that emotionally activate them and short-circuit their reasoning, things "close to the bone" (things they either love, hate, or fear). It's important to note that these trigger topics are often rooted in deep, understandable, and often positive instincts -- patriotism, group bonding, protecting children, etc. These chess pieces are already on the board.

Once mapped, social subgraphs can then be analyzed to find supernodes -- the accounts (real or bot) that are best at producing highly viral content in each subgraph. These accounts can then be used to seed specific content that gently starts to map trigger topics to target belief outcome -- to *condition* people, in the classic, operant conditioning, Pavlovian sense.

By aligning trigger topics with belief outcomes slowly over time, target populations can be pushed along a spectrum of beliefs until their behavior seems quite at odds with their beliefs.

Some populations are more vulnerable to this than others, either by nature or nurture. To exploit this, subgraph data, including influence outcomes, can be mapped to demographic data (age, race, gender, political or religious affiliation, buying habits, data etc.), as well as data from public leaks. This correlation enables automation of identifying, and efficiently focusing effort on, vulnerable populations.

The element of surprise is also important. Weaponizing these subgraphs takes time ... and can be countered if spotted too early! So this infrastructure is also used to study which methods are *more likely to not leak out too soon*. (Private chat groups, on Facebook and elsewhere, seem efficient here.)

Another important element is inoculation: conditioning target populations to ignore, and even be actively hostile to, factual talking points that might otherwise persuade a reasonable person. (This is why reactions to such talking points can be startling immediate, automatic, homogeneous, and dismissive. It may also be a factor in why polling is misleading; it seems feasible to condition people to avoid, or even actively lie to, pollsters).

Taken in total, it should be clear why externally observable outcomes might seem inexplicable -- and why outsiders dipping into these input streams can find it so disorienting and self-contradictory. By the time visible markers of this influence start to "bubble up", the Overton window of what its victims will believe has already dramatically shifted. Put another way: when a public figure starts dropping specific talking points to their base that seem instantly popular "out of the blue" ... it was only out of the blue to you. That topic was a submerged iceberg -- one that the target population has been exposed to for months or years. Only when the benefits of exposing the tip of that iceberg are judged to be worth the reveal, will it be "burned" (to mix my metaphors).

It should also be clear why traditional reasoning about root causes ("What caused this?" punditry) seems to keep falling short. By minimizing or skipping entirely the role of disinformation, think pieces and news coverage trying to leverage pre-social-media concepts to grapple with unexpected outcomes are woefully missing the mark. It should be obvious that part of the reason for this is that they have no tools to observe or assess this influence.

This should also help to explain that while protecting the voting process itself is necessary, it is not sufficient; and why relatively little money might need to be spent on campaigning:

There is less need to hack the vote ... if you've already hacked the voter.

The tools of digital marketing have been repurposed to weaponize your friends, co-workers, and family, in an almost Manchurian Candidate way. Which -- to me personally -- means that we should not be fully blaming the victims here. Were many of them already like this? Probably; exploiting the subgraph efficiently means discovering and exploiting the pieces that were already on the chessboard. But would those people have taken things to this extreme without amplification at scale? Probably not.

The people are on the outside, and the machine is on the inside.

Some questions:

Who is running the machines?
How long ago did AI get applied to accelerate them?
What are the visible signs of machines acting in tandem, or in opposition?
Assuming it's true ... what can we do about it?

References:

The War on Pineapple (CISA)

‘The Great Hack’: Cambridge Analytica is just the tip of the iceberg (Amnesty International)

Sort by Controversial (Scott Alexander)

Fixing "error code: 1275" when using PHLASH to update BIOS on ancient laptops

noreply@blogger.com (Royce Williams (TychoTithonus)) — Sun, 17 May 2020 05:27:00 +0000

If you happen to be turning old laptops into Chromebooks or something, and you're trying to flash the BIOS on an ancient laptop, and you can't do it from a DOS boot USB because it's a Windows-only flash utility, and you run into something like this:

Cannot load driver C:\dell\BIOS\WINPHLASH\PHLASHNT.SYS.

Please check your accounts, if you have no administrator privilege, please login again!

This driver has been blocked from loading

Error code: 1275

... then you're not alone.

Here's what I tried that didn't work:

~~Booting from a Windows 7 universal install disc~~. Trying to run PHLASH threw the error "the subsystem needed to support the image type is not present". I suspect this is because the install disc is primarily 64-bit, but PHLASH is 32-bit, but that's just a guess.
~~Installing 64-bit Windows 7~~. The driver wouldn't load, throwing "Access is denied" in PHLASH.LOG, even when running as true Administrator. My theory is that this is because a newer kind of library or approach is needed by Windows 7 to write to the BIOS while Windows is loaded, and newer versions of the OS block the old method.
~~Installing Windows XP~~. I couldn't find a version that had a variant of the offline algorithm used to validate the license key that recognized the key that the laptop actually shipped with. If I'd figured out how to do this, I suspect that it might have worked.

What did work was to boot from a 32-bit Windows Vista install/rescue CD. PHLASH was able to successfully run from there.

Digging up an ancient BartPE might also work.

Updating the BIOS didn't fix my actual problem (some of the Neverware CloudReady menus are entirely black, as if there's some kind of video-driver overlay problem).

But at least this Inspiron 1501 now has a 2008 BIOS instead of a 2005 one. "I'm doing my part!"

TIL: the unexpected back story of Mah Nà Mah Nà

noreply@blogger.com (Royce Williams (TychoTithonus)) — Sun, 20 Oct 2019 02:36:00 +0000

Just read the Wikipedia article and the 2011 Slate piece.

... and then enjoy them:

The original, "Mah Nà Mah Nà", from the 1966 Italian movie "Svezia, inferno e paradiso" ("Sweden: Heaven and Hell")

The original 1969 Sesame Street version (video)

The 1971 Benny Hill version (audio only)

The version from 101st episode of the Muppet Show (video)

The 2011 Muppets Tonight sketch parody, featuring Sandra Bullock

The entry on the Muppet Wiki - many, many differnent versions and references

(Interestingly, the scat sequences actually vary from version to version, borrowing from a variety of musical sequences.)

Notes on passphrases as passwords

noreply@blogger.com (Royce Williams (TychoTithonus)) — Sat, 31 Aug 2019 22:57:00 +0000

As we promote passphrases, we need to make it clear that not all phrases are created equal.

Phrases that match the rhythm of human language, and phrases that are even only somewhat based on anything that has already appeared anywhere -- lyrics, song titles, movie quotes, phrases that appear on Wikipedia, etc. -- are subject to guessing. The password cracking community is hot on this trail, and really pushing the envelope in this area. "Myd0ghasfle@s!" is certainly better than "abcdefgh" or "Summer2015!", but it is not sufficiently resistant to state-of-the art attacks. These attacks can quickly combine phrase lists with transformational rules, and really tap into the psychology of how people select naive passphrases.

Put another way ... we need to make sure that users do not walk away thinking that "itsmypartyandillcryifiwantto" is a good passphrase.

One problem is the word "passphrase". The layperson expects something called a "passphrase" to be a real, grammatically correct phrase, but that's exactly what we don't want them to use. Such phrases are inherently easier to guess. Only randomly-selected words provide resistance to both brute force and rules-based attacks.

The best balance of "memorizability" and strength is to use "Diceware"-style passphrases, just as that XKCD suggested (but perhaps with some adjustments to the size of the word pool and number of words). The words need to be truly randomly selected, and drawn from a relatively large pool of words that are nevertheless familiar enough to be easily remembered. An open-source, client-side, JavaScript implementation is available here, and some great statistics about the strength of the resulting passwords are shown there.

The sheer numbers behind Diceware-style passphrases are significant. When implemented correctly, even if I tell you which dictionary I've drawn from, and how many words that I used, brute-force cracking is still infeasible for any modern (and even many retired/outdated) password hashing methods.

One implementation hitch is that many sites either A) still apply naive complexity requirements to very long passwords, or B) don't allow passwords longer than a certain length.

To make Diceware-style passphrases compatible with naive complexity rules, users can add their own additional rule. Such a rule can make randomly generated passphrases compliant with such complexity rules. For example, I can always capitalize the first word, and appending "1!" at the end ("Correct horse battery staple1!") Even if I tell someone what my personal additional rule is, brute-force cracking is still infeasible.

If a website has an artificially low length cap (such as 15 or 20 characters), rather than trying to cram a shorter passphrase in, I recommend falling back to a randomly generated password that maxes out the allowed length, and storing it in a password manager.

The best things that web developers can do to support good passphrases is to:

relax complexity requirements once a sufficient length is met, and
support a high maximum password length (such as 128 characters)

Now with the SSLs

noreply@blogger.com (Royce Williams (TychoTithonus)) — Wed, 28 Oct 2015 13:30:00 +0000

This blog (along with all other blogspot.com blogs) is now available over HTTPS:

https://roycebits.blogspot.com/

The RSS and Atom feeds are also HTTPS-aware:

https://roycebits.blogspot.com/feeds/posts/default?alt=rss
https://roycebits.blogspot.com/feeds/posts/default?alt=atom

Please update your bookmarks and feeds accordingly.

Geek note: you can see the Qualys SSL Labs Server Test results for this new setup. (Not perfect, but it's progress!)

Note: HTTPS is not yet available when wrapping a blogspot.com blog within your own domain (such as blog.example.net).

Hash filtering - an appeal to more than vanity

noreply@blogger.com (Royce Williams (TychoTithonus)) — Wed, 28 Oct 2015 03:09:00 +0000

tl;dr: The major password-cracking projects should add support for general hash substring search (vanity hashes, partial collisions, etc.).

What is a vanity hash?

A vanity hash is a hash that contains an interesting substring. For example, my actual Bitcoin address is:

1AzVVHTijMGoWzPtUJoWkF5u6vyS1NcTzj

But it would be cooler if it was this:

1RoycehyNKfRC4nN5jKruZpjMKwNviQRb

That's a vanity hash. Finding one requires a brute-force search. The more characters you want, the harder it gets -- exponentially harder. But even just a few characters can be good enough.

Current examples of vanity hashes

We humans are vain creatures -- so when hashes are used as public identifiers, we want to personalize them:

Bitcoin addresses can be "vanity addresses".
Onion hashes and GPG keys can be vanity hashes.
Tripcodes (used to prove identity on image boards) have vanity hash tools that can search for descrypt-like vanity hashes (at speeds in the 950Mh/s range!)
Git commit hashes can be vanity hashes if patched to support it.
Some folks want to personalize certificate signatures.

More than vanity

Searching for a vanity hash is really just an attempt to find hashes with certain content. Other efforts seek hashes with specific content for other reasons - hash collisions, partial collisions, etc. More generally, these are searches for hash substrings.

Why hash substring search is interesting

Finding substrings in hashes isn't just about vanity. Bringing hash substring search into the cracking platforms would enable other interesting work:

Experimenting with collisions and partial collisions. This is a potential attack surface that is not adequately publicly explored. Nation-states are almost certainly capable of locating near-collisions for some hashes. When you check a download hash, how often is that check only a cursory visual check? True collisions for slow hashes are nigh-infeasible, of course -- but near-collisions may be interesting.
Demonstrating weaknesses. For example, short GPG key IDs can have collisions.
Filtering for hashes with broad properties (instead of just substrings). Hashes that contain only digits, or only letters, or letters and digits alternating, or only lower case (for hashes that distinguish case). Perhaps only a curiosity today, but they might be interesting in the future.
Automatic/loose discovery of similar strings. This would be a nice-to-have at some point, but searching by edit distance would complement substring searching nicely. A minimum or maximum edit distance could be specified to find strings that are "close enough". One of the most common measurements of edit distance is Levenshtein distance, which has been implemented in most languages, and implemented efficiently on GPU.
Working with non-contiguous substrings. Most vanity-hashing tools won't let you search for a hash that begins with "[aA][bB][lL][eE]" and ends with "[eE][lL][bB][aA]". But password crackers' mask syntax is perfect for this.
Unforeseeable innovation. Once this capability is available across all hash types, people will tinker with it, revealing new use cases. I've seen some CTFs that involve searching for specific substrings in hashes. I think they're on to something.

Calling them "vanity hashes" doesn't do justice to the potential. I think that the activity of hash substring search -- partial collisions, vanity hashes, etc. -- needs a better name.

Let's call it hash filtering.

Why the password crackers are the best place to do hash filtering

It's efficient. No one is better positioned to implement hash filtering than the password crackers -- and doing it anywhere else is a waste of resources. Each standalone vanity-search implementation reinvents the hash bruteforcing wheel - usually poorly: This rarely approaches the speeds that JtR and hashcat are capable of (though there are exceptions). The Bitcoin folks have an edge today because of FPGAs, but I expect this gap to close.
It's high leverage. If implemented as a general framework within password crackers, all current and future hash types automatically inherit substring search capability, with little additional effort.

How the password crackers might respond

Sound great! That would be awesome, but not likely to be the first reaction. So ...
OK -- but it should be done as a separate executable. This is a waste of time. The best features of professional-grade password crackers -- core brute-force power, Markov, forking/parallelism, session management, masks, device selection, wordlist management, etc. -- would either be unnecessarily duplicated, left out, or suffer from bit rot.
No. It will slow down cracking too much. Not necessarily. Hash filtering should be implemented as an early pre-optimization step, so that you only have to take the inevitable speed hit if you want hash filtering. Regular cracking would be unaffected.
No. We're not interested in vanity hashing as a concept. That would be a mistake. Vanity hashing is already driving interest, contribution -- and even innovation (like entirely new S-boxes).
No. It might attract noobs. Unavoidable -- but this should be offset by an influx of insight and talent. Also, we have to feed a certain number of noobs to Xanadrel, the Sarlacc of #hashcat. ;)
No. It adds complexity. Technically true, but it should be a largely one-time cost. Also, I think that implementing Levenshtein distance on GPU would be kinda fun if I was atom or Sayantan. ;)
No. Hash filtering itself will be too slow. By its nature, it will, indeed, be much slower than password cracking. But if it can be pre-optimized (see previous point), then it can take advantage of the rest of the cracking support structure, so that hash filtering will be faster, more portable, and easier to use than it would be anywhere else.

Implementation suggestions

Allow the user to specify a "hash filter", using existing mask syntax. This would automatically enable specifying one or more desired substrings anywhere in the hash -- beginning, end, or middle -- as well as both non-contiguous strings and loose matching by character set.
On the command line, consider long options with names like --hash-filter [mask] and --edit-distance [integer]. Or name it something else -- but keep it consistent across projects.
Implement in the pre-optimization pass, so that the slowdown introduced by examining each hash will only kick in when hash filtering is explicitly requested.
Add simple sanity checking. If the user supplies a mask that is longer than the target hash, warn rather than silently truncating. Also, warn if the user wants to filter for a mask that isn't possible (for example, if their descrypt filter's last character is not within [.26AEIMQUYcgkosw]).

Potential bounty?

If this proposal is not obviously compelling, I will consider setting up a bounty (or charitable donation of the winner's choosing). The bounty would go to each major natively Linux-based project (John the Ripper or hashcat) that incorporates hash filtering. Edit distance would be optional.

If you want to go in on a bounty, contact me and I'll do the Bountysource setup.

The ask

Bring vanity hashing and its kin into the fold -- generalize it into hash filtering that can be applied to all current and future hashes. I think that the results will bear interesting fruit.

Updated: to use a valid example Bitcoin vanity.

Cyber and information security certification frequency in job descriptions - 2015-10 edition

noreply@blogger.com (Royce Williams (TychoTithonus)) — Wed, 21 Oct 2015 02:48:00 +0000

A rough study of Indeed acronym frequency when coupled with the word "security":

CISSP  12859 ISC main
ITIL  7233 see also ITILv3 below (+162)
CISA  5374 ISACA auditor
CISM  3927 ISACA manager
CompTIA 2857 a+, security+, network+, etc
GIAC  2368 (any)
CEH  2329 EC-Council
GSE  1501 GIAC expert/advanced
SSCP  1328 ISC – CISSP “lite”
GCIH  1279 GIAC incident handling
CRISC  804 ISACA risk
“CCNA Security” 660 Cisco
GPEN  554 GIAC pentester
OSCP  475 Offensive Security
GSLC  332 GIAC's CISSP analog
GWAPT  253 GIAC web pentest
“CCNP Security” 235 Cisco
CCSE  228 Checkpoint
GREM  219 GIAC reverse engineering malware
ITILv3  162 See also ITIL above (+7233)
“CCIE Security” 155 Cisco
GCFE  140 GIAC forensic examiner
OSCE  124 Offensive Security – advanced
CSIH  100 Carnegie Mellon
GXPN  73 GIAC exploit dev and advanced pentest
GCWN  66 GIAC Windows security
“Cisco CSS” 29 Cybersecurity Specialist – hard to search for right
WCAN  4 Wireshark

No sense in wasting the heat and air movement

noreply@blogger.com (Royce Williams (TychoTithonus)) — Fri, 28 Aug 2015 16:26:00 +0000

My 6x GTX 970 open-air password-cracking rig.

Purpose:

On-site password auditing under my consulting shingle, Tech Solvency
Independent password research

Components:

Case: Custom Spotswood
- Case cover: Not pictured. Mostly for transport purposes, usually runs with the cover off
- Handle: For carrying. Sorta helpful, but a little unwieldy
Board: GA-990FXA-UD3 rev 4.0. Leave IOMMU enabled; add "iommu=soft" to grub parameters (otherwise, USB and NIC won't work)
- Sound: No onboard sound - mini onboard speaker ($4)
CPU: AMD FX-8350
Memory: 32G for PRINCE headroom
OS: Ubuntu Server 14.04 LTS 64-bit
PSUs: 2x Corsair RM1000. No fan movement unless loaded. Room for growth if cards get upgraded
- Dual PSU adapter: Vantacor
- Power switch: Had to get a power/reset switch/cable - none onboard
GPUs: 6x EVGA 04G-P4-2974-KR. Max 165W per card
- Risers: Non-USB (ribbon) risers. The four x16 slots have powered x16-to-x16 ribbon risers, which isn't necessary for most GPU-centric cracking. x4 or higher is better for John the Ripper, but x1 is fine for cudaHashcat. The two x1 slots have unpowered x1-to-x16 ribbon risers. The USB-based risers would *not* work with this board. The symptom was that only a couple of the cards would work at a time.
- GPU fans: locked at 50%. Temperatures do not break 78 at normal house temps, and do not break 72 in cooler rooms

Power consumption:

Idle: 105W
Idle, with GPU fans forced to 50%: 120W
Full CPU load (all 8 cores, OEM CPU fan at 3600 RPM): 265W
No CPU, oclhashcat job loaded but paused: 310W
Full GPU-only oclhashcat job in progress, minimal CPU: 1050W
Full processing load (CPU and GPU): TBD

UPDATES - Performance:

UPDATE - upgrades over time:

GPUs are now 6x 1080s
Ubuntu 18.04 LTS
CUDA 10.1 (upcoming hashcat 6.0 restores CUDA support; now you can use all of your GPU memory! - good howto from Puget Systems here)
almost always latest hashcat from GitHub

The E-mail Productivity Curve

noreply@blogger.com (Royce Williams (TychoTithonus)) — Fri, 19 Jun 2015 13:34:00 +0000

Brilliant. Captures exactly why we hate email -- and why we love it.

Via: The E-Mail Productivity Curve - Cal Newport

The origins of the Olaf card

noreply@blogger.com (Royce Williams (TychoTithonus)) — Fri, 06 Feb 2015 20:00:00 +0000

Late one night, I was telling my friend Sam that I had recently run into a panhandler who was deaf, and who communicated via a stock message on a business-sized card. Having someone wordlessly hand you a card was a strangely powerful experience. Sam and I immediately started thinking of alternate messages that could be conveyed with this new medium. Somehow, we latched on to the idea of a total stranger using it to ask people to tickle them. And it would be even funnier if English wasn't their first language. This quickly evolved into a large, friendly Norwegian man, asking to be tickled:

("Hello, My name is Olaf. I am recently from Norway and do not speak the English good. Please tickle me. Thank you.")

The idea was to walk up to a total stranger and hand them the card– without saying a single word. The wordlessness is a crucial element. Your initial communication with the person is solely through the card – and your friendly, hopeful facial expression.

We laughed until our sides hurt.

Then, in classic Sam style, he insisted that we go to Kinko's at once – at 1 o'clock in the morning! – to have the cards actually printed. My friend Mike Hanscom was working at that Kinko's, and was delighted to oblige, chuckling the whole time.

The first run was 500 cards. We burned through that one pretty quickly. The second run of 500 went fast, too. The fad faded in the middle of the third batch, but a few stories stand out.

I have almost never handed them to total strangers. Instead, I show them to people whom I already know (or have just met), saying, “Pretend like you don't know me, and I walk up to you without saying anything, and I hand you this card.” Since I look faintly Scandinavian, and I've usually just met them, it's not too much of a stretch – and it was a fun way to break the ice.

But Sam actually took a batch of cards to Costco to perform a sociological experiment. (If you don't know Sam, imagine a 6-foot-5 cross between Kyle MacLachlan and Waldo from “Where's Waldo.” When he walks up to you as a total stranger and hands you an Olaf card, you're going to pay attention. Sam handed cards out to people until the Costco folks asked him to leave. He had enough data to identify three major categories of response:

85% of people would laugh, take the card, and keep walking.
10-12% would immediately turn and go without any response at all – studiously making no eye contact.
The remaining 3-5% would look at Sam furtively, look down at the card, look at Sam again … and then reach out very gingerly, tickle him very briefly, and then make a break for it.

Sam's theory was that this last group was afraid of what he would do if they didn't tickle him.

Sam and I knew that we had struck some kind of chord when we took a road trip to UAF and saw one taped to someone's dorm door – someone whom neither of us knew.

The largest distributor was my friend Rod, who moved to Tuscon and started handing them out. He would go out dancing in his vintage green '70s leisure suit, giant afro wig, and big sunglasses ... and hand out Olaf cards. He went through an entire batch himself. For many in Tucson, Rod is “Olaf.”

Rod was also a dinner captain at a very nice steak restaurant in Tucson. One night, Kevin Spacey had been a customer and was on his way to the door when Rod intercepted Spacey briefly and handed him an Olaf card. As Spacey was walking away, out of the corner of Rod's eye, he saw Spacey look down at the card, actually read it, chuckle, and then put it in his pocket. High praise, indeed.

To this day, every once in a while, I'll get a “hey, you're the guy who handed out the Olaf cards!”

Managing and optimizing lists of password masks

noreply@blogger.com (Royce Williams (TychoTithonus)) — Sun, 25 Jan 2015 23:52:00 +0000

I've been working on some password-cracking research on the side. I thought I'd come up with a cool new idea, but it turns out that someone else already thought of it.

It occurred to me last night that a big list of passwords could be abstracted out into their equivalent masks, and then a frequency count of those masks could be generated, which could then be exhausted in frequency order.

First, I extracted a frequency count of character set combinations (masks) from all eight-characters-longthe RockYou breach's password list, yielding a list of the form:

100:hundredofthese
95: 95ofthese
[...]
2:justtwoofthese
1:onlyoneofthese
1:alsoonlyoneofthese

... as follows:

#!/bin/bash

echo "- Getting frequency of character patterns from RockYou ..."
time gunzip -cd rockyou.txt.gz \
        | tr '[:lower:]' 'l' \
        | tr '[:upper:]' 'u' \
        | tr '[:digit:]' 'd' \
        | tr "[\ !\"#$%amp;&\'()*+,-./:;<=>?@\[\\\]^_\`{|}~]" 's' \
        | sed 's/[^luds]/a/g' \
        | strings \
        | cut -b1-8 \
        | freqcount \
        > rockyou.freq.8a
wc -l rockyou.freq.8a
head rockyou.freq.8a

echo "- Generate masks."
echo "- Ignoring all masks with more than three consecutive 'a' charset."
time cat rockyou.freq.8a \
        | cut -d\: -f2 \
        | sed 's/l/?l/g;s/u/?u/g;s/d/?d/g;s/s/?s/g;s/a/?a/g' \
        | egrep -v 'aaaa' \
        > rockyou.masks.8
wc -l rockyou.masks.8
head rockyou.masks.8

echo "- Done."
#end of script

Next, I wrote a script to exhaust each one in order by frequency using hashcat:

#!/bin/bash

for mymask in `rockyou.masks.8`; do
        echo "- Running mask: $mymask ..."
        cudaHashcat64.bin -a 3 -m 1500 \
                target-hashes.list \
                $mymask
        echo "$mymask: done - `date`" >> $0.log
done
#end of script

Then it occurred to me that if someone else had published this info, and had used real corpora of passwords as the input, then our frequency lists would probably look similar. So I did the following Google search:

"?l?l?l?d?d?d?d" "?l?l?l?l?l?d?d?d"

... and the first hit was the KoreLogic blog post.

Dangit! :-) But at least I'm catching up to the state of the art; the KoreLogic article was published in April 2014. :-)

I got the idea from work I had done on some license-plate-collecting stuff I do on the side. I thought of it for capturing high-level patterns in serials, so that people can search for a plate based on the serial. A plate with "BDT 606" on it would match any plate whose serial "mask" is "AAA 999" using my notation. (I then match more closely, but it's used for a high-level search first).

I haven't watched the KoreLogic presentation yet, but I can definitely improve upon my own approach, because I'm being overly aggressive in turning then entire set of non-alphanumeric-but-printable characters into 's':

        | tr "[\ !\"#$%&\'()*+,-./:;<=>?@\[\\\]^_\`{|}~]" 's' \

... when most folks use the simple ones (#$%@, etc.) I could create a custom charset for this using the notation as noted here ... and then turn the remaining characters into another custom charset that is the remaining characters.

I then found PACK - the Password Analysis and Cracking Kit, which is is a set of Python scripts to manage masks, including optimizing a set of masks based on a given timeframe (or, "I have 24 hours. Which masks should I use to maximize how many passwords I can crack?")

FreeBSD LSI SAS9211-8i HBA firmware notes

noreply@blogger.com (Royce Williams (TychoTithonus)) — Sun, 25 Jan 2015 23:43:00 +0000

I'll be using this post to store information about LSI HBA firmware, with a focus on FreeBSD (but also drawing upon Linux information). It may also be useful for users of FreeNAS, PC-BSD, unRAID, Nexenta, or ZFSguru.

Why - SATA port density on a budget

If you are using ZFS, you do not need RAID -- you just need lots of fast SATA ports. To maximize the features of ZFS, it needs to directly access attached drives in JBOD mode rather than as RAID. If you can afford them, you can buy the LSI 9211-8i HBA card. Alternatively, you can also buy a less expensive card, and then replace its stock "IR" (Initiator-RAID) firmware by "crossflashing" to an "IT" (Initiator-Target) version of LSI's general firmware for 9211-8i hardware. This option is useful for people building home NAS systems on a budget. Popular cards include the Dell PERC H200 and the IBM ServeRAID M1015. This ServeTheHome post introduces the topic well.

Here is the relevant dmesg for a Dell PERC H200 Internal (H200I) under FreeBSD 8.4-RELEASE. (Note that this particular card's LSI firmware (Phase 9) is out of sync with the FreeBSD driver (Phase 14), which may have unexpected side effects. The system was initially built as a FreeBSD 8.1-RELEASE system in 2010.)

$ uname -r
8.4-RELEASE-p19
$ egrep ^mps0 /var/run/dmesg.boot
mps0: <LSI SAS2008> port 0xc000-0xc0ff mem 0xfb3b0000-0xfb3bffff,0xfb3c0000-0xfb3fffff irq 16 at device 0.0 on pci3
mps0: Firmware: 09.00.00.00, Driver: 14.00.00.01-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: [ITHREAD]

General flashing tips

Before flashing, and especially before erasing any flash, use the sas2flsh.exe -listall option to note the SAS ID of your device (usually beginning with "0x590"). If you accidentally erase the entire flash (sas2flsh.exe -o -e 6 will retain your SAS ID, but sas2flsh.exe -o -e 7 will wipe it), you will not be able to re-flash the device unless you have this ID. Write it down.

Some earlier versions of sas2flsh.exe allow cards to be flashed from IR firmware to IT firmware; others do not. I and others have had luck with the one that comes with LSI's Phase 7 (AKA P7 or P07) firmware. (Try this link, or search LSI.com for "9211_8i_Package_For_P7_Firmware_BIOS_Upgrade_on_MSDOS_and_Windows" to download the package that contains this version of sas2flsh.exe.

To flash the firmware on cards installed in non-UEFI motherboards, you can create a DOS-bootable USB key using a tool like Rufus. Rufus will make the device bootable with FreeDOS or MS-DOS (well, actually, Windows ME!). I and others have had better luck using the MS-DOS option. (According to that thread, LSI themselves recommend MS-DOS rather than FreeDOS).

Also note that when flashing using sas2flsh.exe there are two different components to be flashed: the firmware (contained in a filename sometimes ending with .fw, and usually named after the device in some way) and the BIOS (usually named something like MPTSAS2.ROM). The firmware component is what your OS driver communicates with. The BIOS component allows you to configure the firmware at boot time, and can enumerate the list of attached hard drives. For ZFS and JBOD purposes, the BIOS is not strictly necessary, and has even been reported to cause problems when present. Erasing the firmware areas sas2flsh.exe -o -e 6 and then just applying the firmware without the BIOS will also result in faster boot times.

A common error that people get when flashing is "Failed to Validate Mfg Page 2". This occurs when you try to flash to the LSI firmware without first erasing the firmware. The techmattr blog has some good information.

Phase 10 firmware or higher is needed for cards in this family (6GB/s HBSa) in order to support drives larger than 2GB. See this LSI KB article (old version cached at the Internet Archive)

FreeBSD flashing considerations

At this writing (2015-01), there have been reports of Phase 20 not playing well with FreeNAS and FreeBSD. Downgrading to Phase 16 (FreeBSD 9.3 and 10.0) or Phase 19 (FreeBSD 10.1) is reported to be more stable.

Under FreeBSD, PC-BSD, and FreeNAS, the desired end state is for the "Firmware" and "Driver" ports of the dmesg line to use identical firmware versions. For FreeBSD 10.1-RELEASE, this is the Phase 19 version. In the dmesg output, the Firmware item is what's on the card, and the Driver item is what the OS supplies.

mps0: Firmware: 19.00.00.00, Driver: 19.00.00.00-fbsd

In fact, FreeNAS will even complain if they are mismatched.

(I also list all of the firmware/OS pairings I know of towards the end of this post.)

Beware when upgrading a FreeBSD-based OS. Depending on the combination of firmware and driver, your drives may disappear from the OS' view until you reflash. This can be especially troublesome if your root filesystem is ZFS.

How to reflash the Dell Internal Tape Adapter 15MCV card as a 9211-8i

There is a card from Dell that looks almost identical to the H200I card, but is actually a Dell Internal Tape Adapter board (Dell part number 15MCV). This is identified in various levels of firmware and utilities as "Int Tape Adapter" or "IntTapeAdptr", and identified under Linux as:

 Vendor(0x1000), Device(0x0072), SSVID(0x1028), SSDID(0x1F22)

Cards labeled as "H200" on eBay are sometimes actually these cards instead. Unfortunately, the usual methods for flashing to generic LSI drivers do not work for the Tape Adapter boards. But as discovered by Hardforum user lamune in this post, if you start from the original Dell Internal Tape Adapter firmware, and then, without erasing the current firmware, flash using Supermicro HBA drivers (Phase 16 at this writing) as an intermediate step, you can then flash to the LSI firmware.

Here is the Linux dmesg for my Internal Tape Adapter board, prior to being cross-flashed. Note that capabilities include RAID, and the BIOS has a standard version (07.11.10.00):

$ dmesg | egrep -i 'lsi|mpt|mps|sas'
[    0.000000]   HighMem  empty
[    5.722377] mpt2sas version 16.100.00.00 loaded
[    5.731608] scsi4 : Fusion MPT SAS Host
[    5.739575] mpt2sas0: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (497212 kB)
[    5.739643] mpt2sas 0000:01:00.0: irq 43 for MSI/MSI-X
[    5.739682] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 43
[    5.739686] mpt2sas0: iomem(0x00000000dfcb0000), mapped(0xe0280000), size(65536)
[    5.739689] mpt2sas0: ioport(0x000000000000dc00), size(256)
[    6.028016] mpt2sas0: sending diag reset !!
[    7.268013] mpt2sas0: diag reset: SUCCESS
[    7.418064] mpt2sas0: Allocated physical memory: size(4134 kB)
[    7.418070] mpt2sas0: Current Controller Queue Depth(2748), Max Controller Queue Depth(2879)
[    7.418073] mpt2sas0: Scatter Gather Elements per IO(128)
[    7.648484] mpt2sas0: LSISAS2008: FWVersion(07.15.08.00), ChipRevision(0x03), BiosVersion(07.11.10.00)
[    7.648490] mpt2sas0: Dell 6Gbps SAS: Vendor(0x1000), Device(0x0072), SSVID(0x1028), SSDID(0x1F22)
[    7.648492] mpt2sas0: Protocol=(Initiator,Target), Capabilities=(Raid,TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[    7.648577] mpt2sas0: sending port enable !!
[   10.168254] mpt2sas0: host_add: handle(0x0001), sas_addr(0x590bxxxxxxxxxxxx), phys(8)
[   15.296010] mpt2sas0: port enable: SUCCESS

Here is a Linux dmesg after successful crossflash of firmware, but skipping installing a BIOS. Note that capabilities no longer include RAID, and BIOS is empty (00.00.00.00)

$ dmesg | egrep -i 'lsi|mpt|mps|sas'
[    0.000000]   HighMem  empty
[    5.784639] mpt2sas version 16.100.00.00 loaded
[    5.789191] scsi4 : Fusion MPT SAS Host
[    5.793970] mpt2sas0: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (497212 kB)
[    5.794039] mpt2sas 0000:01:00.0: irq 43 for MSI/MSI-X
[    5.794081] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 43
[    5.794086] mpt2sas0: iomem(0x00000000dfcb0000), mapped(0xe0140000), size(65536)
[    5.794088] mpt2sas0: ioport(0x000000000000dc00), size(256)
[    6.235645] mpt2sas0: Allocated physical memory: size(4964 kB)
[    6.235652] mpt2sas0: Current Controller Queue Depth(3307), Max Controller Queue Depth(3432)
[    6.235654] mpt2sas0: Scatter Gather Elements per IO(128)
[    6.468421] mpt2sas0: LSISAS2008: FWVersion(19.00.00.00), ChipRevision(0x03), BiosVersion(00.00.00.00)
[    6.468429] mpt2sas0: Dell 6Gbps SAS: Vendor(0x1000), Device(0x0072), SSVID(0x1028), SSDID(0x1F22)
[    6.468432] mpt2sas0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[    6.468517] mpt2sas0: sending port enable !!
[    8.978905] mpt2sas0: host_add: handle(0x0001), sas_addr(0x590b11c017d2a400), phys(8)
[   14.116010] mpt2sas0: port enable: SUCCESS

Known FreeBSD versions and their equivalent target mps driver versions

8.2-RELEASE: Phase 12? - not sure, but likely 12.00.00.00-fbsd - did not ship with, but it can be backported
8.3-RELEASE: Phase 13 - mps0: Firmware: xx.xx.xx.xx, Driver: 13.00.00.00-fbsd
8.4-RELEASE: Phase 14 - mps0: Firmware: xx.xx.xx.xx, Driver: 14.00.00.01-fbsd - LSI P14 firmware
9.1-RELEASE: Phase 14 - mps0: Firmware: xx.xx.xx.xx, Driver: 14.00.00.01-fbsd
9.2-RELEASE: Phase 14 - mps0: Firmware: xx.xx.xx.xx, Driver: 14.00.00.01-fbsd
FreeNAS v ?? Phase 15 - mps0: Firmware: xx.xx.xx.xx, Driver: 15.00.00.00-fbsd - LSI P15 firmware (ref)
9.3-RELEASE: Phase 16 - mps0: Firmware: xx.xx.xx.xx, Driver: 16.00.00.00-fbsd - LSI P16 firmware
10.0-RELEASE: Phase 16 - mps0: Firmware: xx.xx.xx.xx, Driver: 16.00.00.00-fbsd
Phase 18 was committed but not in a release that I can tell.
10.1-RELEASE: Phase 19 - mps0: Firmware: xx.xx.xx.xx, Driver: 19.00.00.00-fbsd - LSI P19 firmware
10.2-BETA2: Phase 20 - mps0: Firmware: xx.xx.xx.xx, Driver: 20.00.00.00-fbsd - LSI P20 firmware (reported by Dan Langille)
10.3-RELEASE: Phase 20 - mps0: Firmware: xx.xx.xx.xx, Driver: 20.00.00.00-fbsd - LSI P20 firmware
11.0-RELEASE: Phase 20? - mps0: Firmware: 20.00.07.00, Driver: 21.01.00.00-fbsd - (The FreeBSD driver is version 21, but the latest Avago firmware download (from 2016-04) still shows Phase 20 as the most recent).

Note:After Avago bought LSI, their new download system sometimes makes direct linking more difficult. This search may help.

FreeBSD firmware installers

I haven't had good luck with these, because they often can only perform a small subset of the actions necessary to upgrade firmware. But if you need them, here they are.

(For all firmware and installers, see LSI's archive)

Useful command-line snippets

sas2flsh -listall
sas2flsh -c 0 -o -testssid 1028:%SSID% > SSID.out
sas2flsh -c %num% -f fwname.fw > flash.out
sas2flsh -c %num% -b mptsas2.rom >> flash.out
sas2flsh -c %num% -b x64sas2.rom >> flash.out
sas2flsh -c %num% -o -reset > reset.out

References

The LSI SAS2IRCU utility can show basic information about your card.
Bryan Vyhmeister's blog post about flashing IT firmware
mycusthelp.info comprehensive guide to many kinds of reflashing scenarios
LaptopVideo2Go forums post listing all cards in the hardware families that may support cross-flashing.
This FreeBSD Forums post has good discussion about the process generally, and how to work with UEFI motherboards while flashing.
FreeNAS thread about mismatching firmwmare and FreeBSD driver versions, and cautioning against P20
PC-BSD bug report asking for driver version to be updated. PC-BSD stated that they will stick with P16 and skip P18.
Dan Langille's blog about flashing IT firmware from UEFI on FreeBSD
FreeNAS thread - general overview of the issues
LSI driver download list for SAS 9211-8i
Zorinaq blog post about port counts, hardware compatibility, and overall great overview
eracks blog post about backporting the mps driver to FreeBSD 8.2
Handy eBay search: "dell" (ucs-71,h200,h200i,u039m,47mcv,65f44,15mcv,015mcv,00RR9J,0U039M) -h200e

Yubikey FIDO error: "A timeout occurred while waiting for a Security Key to be inserted or tapped."

noreply@blogger.com (Royce Williams (TychoTithonus)) — Tue, 28 Oct 2014 06:09:00 +0000

If you're putting a Yubikey FIDO USB U2F key into your Linux machine, and activation eventually times out with this error:

"A timeout occurred while waiting for a Security Key to be inserted or tapped."

... this may be due to ownership of the device.

The fix, via this Yubico forums post:

echo 'KERNEL=="hidraw*", SUBSYSTEM=="hidraw", MODE="0664", GROUP="plugdev", ATTRS{idVendor}=="1050", ATTRS{idProduct}=="0113|0114|0115|0116|0120"' | sudo tee /etc/udev/rules.d/70-u2f.rules Before: crw------- 1 root root 250, 6 Oct 27 21:57 hidraw6 After: crw-rw-r-- 1 root plugdev 250, 6 Oct 27 21:59 hidraw6

Activation should now proceed normally. Some folks had to then also restart Chrome. Chrome 38 or higher required.

Best fake WHOIS business name ever

noreply@blogger.com (Royce Williams (TychoTithonus)) — Thu, 09 Oct 2014 05:42:00 +0000

Best fake WHOIS business name ever:

Admin Organization: LiddyCorp Tofu Mining Corporation of Antarctica

via https://saintaardvarkthecarpeted.com/blog/

A different kind of Rick(man) roll

noreply@blogger.com (Royce Williams (TychoTithonus)) — Thu, 06 Sep 2012 06:03:00 +0000

Worth the wait. Larger version here, and more here.

The project featured an array of glacially paced performances of theater artists and actors all genres and nationalities. With artists featured both singly and in groups, the piece offered a unique and secret glimpse into some of the world’s greatest performing artists.

Via: The Long Now blog

Reframing the acceleration of addictiveness

noreply@blogger.com (Royce Williams (TychoTithonus)) — Tue, 08 May 2012 06:49:00 +0000

Tantek Çelik knocks it out of the park with his The Acceleration of Addictiveness vs Willpower, Productivity, and Flow, putting a more positive spin on Paul Graham's Acceleration of Addictiveness:

Paul concludes his essay with We'll increasingly be defined by what we say no to. He's right, we will and are.

The problem is, "saying no" is expensive. It costs you willpower to do so. Explicitly saying no doesn't scale. We need automatic or default ways to "say no". The best word we have for that is "filters".

... and:

This is your life and it's ending one minute at a time.

The time you spend giving into your urges and supporting your addictions is time you could have spent being creative and productive.

Tantek is collecting practical, constructive information about how to improve the odds on his personal wiki. Great reading.

Smells like a pre-Internet library in here

noreply@blogger.com (Royce Williams (TychoTithonus)) — Tue, 20 Mar 2012 05:36:00 +0000

Here is a list of things that I tend to hoard (where I define hoard as "keep more (or longer than) needed, or conserve or manage more than the effort involved warrants"):

* Alaska license plates (I had to get the obvious one out of the way first)
* books and magazines
* cardboard boxes
* coins
* computers and peripherals
* computer adapters, cords and computer-related tools
* dowls, rods, tubes
* ID cards
* movie stubs
* paper
* pens and pencils
* plane tickets
* twist ties
* water

... and here are the things that I think that my wife hoards:

* books and magazines
* candles
* checks
* coffee cups
* cookbooks
* glass jars
* greeting cards received
* leftovers
* toiletries

Obviously, we have many bookshelves.

APC Masterswitch errors: heap corrupted : non-matching sizes

noreply@blogger.com (Royce Williams (TychoTithonus)) — Tue, 27 Dec 2011 22:42:00 +0000

Sometimes, an APC Masterswitch will start spewing errors like this on its console:

heap corrupted : non-matching sizes :-22904..13041

This sometimes happens when some revisions of the APC Masterswitch OS have too much uptime.

The issue can be temporarily resolved by rebooting the management card. Even though your terminal is filling up with these errors, you can still actually log in and reboot the card - you just have to do it semi-blind.

For some revisions of OS, the key sequence is:


[username]
[password]
3 (the 'System' menu)
5 (the 'Tools' menu)
1 (the 'Reboot' option)
YES (all in caps, to confirm that you want to reboot)

This will clear the error without power-cycling any devices. I suspect that an OS upgrade would address the issue permanently.

Better late than never

noreply@blogger.com (Royce Williams (TychoTithonus)) — Fri, 23 Dec 2011 21:54:00 +0000

I just realized that I didn't mention something important here.

We had a baby!

Using vi key bindings in Perl's debugger on FreeBSD

noreply@blogger.com (Royce Williams (TychoTithonus)) — Sat, 22 Oct 2011 19:55:00 +0000

Even after verifying that Term::ReadKey and Term::ReadLine were part of my perl distribution:


royce@heffalump$ perl -e 'use Term::ReadKey;'
royce@heffalump$ perl -e 'use Term::ReadLine;'
royce@heffalump$

... and making sure that vi key bindings were listed in my .inputrc:


royce@heffalump$ grep editing-mode ~/.inputrc
set editing-mode vi

... I still couldn't use 'em, as demonstrated by what happened when I tried to use movement keys:


  DB<1> testtesttest^[[A

In my research, I discovered that Ubuntu folks were installing a different ReadLine. I eventually found the devel/p5-ReadLine-Perl port, which has this pkg-descr:


Perl 5 ships with a module called Term::ReadLine which is an interface
to command line editing and recall.  The version that ships with Perl
is only a stub, and offers little functionality.

This module supplants the Term::ReadLine stubs with real command line
editing and recall facilities, written entirely in Perl.  Applications
that use Term::ReadLine do not need to be modified to gain the benefits
of this package; it will happen transparently upon installation.

After installing p5-ReadLine-Perl, I'm up and running.

FreeBSD apr1 upgrade error: Configure: 9904: Syntax error: word unexpected (expecting ")")

noreply@blogger.com (Royce Williams (TychoTithonus)) — Mon, 26 Sep 2011 14:15:00 +0000

I was having trouble with apr1 on a FreeBSD web server. apr1 is used by Apache. The configure script for apr1 was dying with this error:


Performing libtool configuration ... 
. / configure: line 9904: syntax error near unexpected token `lt_decl_varnames, '

... which boiled down to:


. / configure: line 9904: `lt_if_append_uniq (lt_decl_varnames, SHELL,,, '

Fortunately, this thread was eventually resolved by someone finding out that they had some libtoo115 files left over, even though it had been deinstalled. I manually removed the extraneous files with:


# pkg_delete libtool-1.5.24 
# rm -rf /usr/local/share/libtool15 
# rm -f /usr/local/bin/libtool15 /usr/local/bin/libtoolize15

I am now back up and running!

Thanks to Vladislav Staroselskiy for a very helpful post about the FreeBSD apr1 libtool15 problem.

A little post-1964-earthquake humor

noreply@blogger.com (Royce Williams (TychoTithonus)) — Fri, 19 Aug 2011 06:17:00 +0000

A guest post from my father, for which I asked him to share a story about something that happened after things had mostly gotten back to normal after the 1964 Alaska Earthquake. Dad worked at what was then the 6981st, and is now the 381st Intelligence Squadron on Elmendorf (now Joint Base Elmendorf-Richardson). For those who know the work, the terminology here will be familiar.

...

The Great Alaska Earthquake happened in late March of 1964. 9.2 on the Richter scale. As many folks know, it was devastating to many parts of southeast Alaska.

Sometime after that event, I was on D Flight “tearing traffic” as usual during a swing shift. One of the Flight’s 292X1s was a “goosey” sort of guy. He was diligently working away that evening as I approached his work station from behind, preparing to “tear traffic” from his position. As I came up behind him I reached up and tapped the fluorescent light fixture hanging directly above. This started the fixture swinging. Then, “tearing traffic” in front of him, I got his attention and looked up as if to suddenly notice the swinging light fixture. He saw I was looking up so he looked up too. He saw the fixture moving and before he had any second thoughts, leaped out of his chair and at double time made for the Operations door. He went past other folks diligently working, through the doors, down the stairs, past the Air Police person guarding access to the upstairs Operations area, through the first floor foyer and out the front doors of the building to the flag pole located in the center of the secure compound area.

Once he got there he couldn’t understand why others weren’t there too. He was sure he had quickly reacted to an earthquake aftershock.

When no one else was around except him and the flag pole it dawned on him that perhaps the swinging light fixture had not caused what he thought. He strolled back into the building, up to the Air Police person on guard duty, showed the guard his badge and continued on up to the second floor and back to his position in the Operations area. He did not stop or even slow down to answer anyone’s questions about his rapid departure a few minutes earlier.

By that time I figured that he had an idea who was responsible for his quick-reaction to the swinging light fixture.

I managed to avoid his attempts to find me through the rest of the swing shift.

Damn 202s! Not funny! Be a takin' 'er easy. Ur Dad sends

Remembering William Sleator

noreply@blogger.com (Royce Williams (TychoTithonus)) — Sun, 07 Aug 2011 15:22:00 +0000

The Sleator family has created a blog for posting memories about William Sleator.

William Sleator, 1945 - 2011

noreply@blogger.com (Royce Williams (TychoTithonus)) — Fri, 05 Aug 2011 14:14:00 +0000

Publisher's Weekly recently tweeted that William Sleator passed away in Thailand on Tuesday, August 2nd. He was 66.

I am a big fan. My default online handle, TychoTithonus, is the name of the main character in his book The Green Futures of Tycho.

I emailed with him a few times, and he signed one of my copies, but I regret that I never met him in person.

His work struck a chord with me in ways that are hard to explain. Some part of my childhood is now written in stone.

Other links about his passing:

Update 2011-08-07: William Sleator's obituary in the New York Times.

Awkward Family Photos - the Board Game

noreply@blogger.com (Royce Williams (TychoTithonus)) — Tue, 12 Jul 2011 20:47:00 +0000

Hmm ... will Michael's famous awkward family photo be included in the new Awkward Family Photos Board Game?