Last week, I started playing around with a project to create my own (Python) site search, including a crawler and Whoosh-based search. I’d seen the implementation of a Lucene search in Zend go fairly easy-peasy, and liked the idea of a self-hosted search.

Problem is–well, one of the problems is–the crawl time for a site with 1200 posts (most of which are low-priority) is a deal-breaker on a shared hosting provider. It takes far longer than 5 minutes just to collect the links, even with multiple threads. Add the parse time to get indexable content for 1200 pages, and I was stuck contemplating how to crawl and index the site in parts.

This sounds like a great, fun, project. …Except that it’s already been done and I have other things I’d rather be doing. Google did it; their index for my site updates surprisingly quickly and doesn’t make me afraid that Dreamhost will smite me. (I’ve been with Dreamhost for several years now, and while I’ve learned how to properly deploy a site since moving here from… Brinkster, was it?, I don’t relish the idea of learning a new environment for all the stuff I run here.)

So instead of the 4-5 hours I’d spent screwing with the Ikea-esque assembly of a site crawler and search, I spent two this week really making Google’s Custom Search Engine (CSE) work for me. Yes, there are ads. Yes, it’s not a solution that I own. (Then again, neither is my email, in that sense.)

What’d you want, Willis?

• A full search page that can give smartly-relevant search results and search quickly.
• A search box on every page that doesn’t jeopardize page load times with blocking JavaScript.
• The sidebar search box to take the user to the search page, rather than doing some AJAX-y search stuff as Google wants to do (in a 200-damn-pixel sidebar, no less).
• Up-to-date search results, so that new entries will appear in the index by the time a new post is made.
• Search results that included the django-driven stories on the site.

I didn’t go as far as uploading custom XML to define the engine for this, but I’ve done some good things here lately to improve page load times, and didn’t want to wreck that by using Google’s standard in-page JavaScript and CSS (Cascading Style Sheets) code snippet.

What it do?

Here’s what I did with my preexisting search engine:

1. Set it to search both “sites” http://irrsinn.net and http://irrsin.net/fiction/*. I found that the stories got buried when I didn’t.
2. Set a few “keywords” for the search engine. I’m not using a Google-hosted search page or results page, so I’m not sure if this matters one fig.
3. Made sure that it’s picking up my sitemap. I use a Python sitemap generator that runs weekly and pushes out my sitemap. Rather than actively crawling my site, it uses my Apache logs to know what to crawl. If it wasn’t visited since the logs cleaned out, then it’s pretty low priority anyway.
4. I set up Refinements. This is kind of slick, and I have no idea if it’ll end up used or not. On my search page now, you’ll see tabs for “Stories” and “Character Sheets” when you’ve searched. Each of those is a search refinement, where I’ve said, “if you’re looking for stories, then I should further filter the results by Transhuman Congress OR Witches of Ming Ung (etc.)”. I see it as a way to guide users either by just reminding them of what they may be looking for or by actively helping them get there. If you search my site for “Vampire” when you’re actually interested in if I’m doing a Vampire character sheet in my initial release, then you can filter away the posts about Greg being a vampire and just see what’s relevant.
5. Chewed on the CSS and JS. More details of this below.

Was the CSS Tasty?

Why yes, it was. I had a couple of beefs here. First, the config panel for search engines doesn’t let you tweak a whole lot of visual options. Second, when you get the code for the engine, all the HTML, CSS, and JavaScript is in one big block. *shudder*

I went ahead and tweaked as much as I could through the config panel, then grabbed the code Google gave me and tossed it into a text file. The CSS went into my template’s style.css, and the JavaScript into my template’s footer. That JavaScript really needs to be in a separate include, so that it can be spliced, minified, and S3-stored with the rest of my JavaScript. The WordPress “Page” that is “/search/” contains one tidbit:

<div id="cse">Loading</div>

The JavaScript (as provided from Google) hooks onto that #cse, so I didn’t have to tweak anything there.

For the CSS itself, I had to use Firebug to grab IDs and classes and style stuff manually. The config panel wasn’t enough for getting rid of the fugly drop-shadows and too-light grey text. When I started making tweaks on the Minimal default style, it created a custom style for me, so I can always reset to the Minimal style if I want to. As I mentioned, all my custom CSS went into my template’s style.css, which is minified and stored on S3.

But Is it Handy?

It is now. The reason I’d previously (and unhappily) removed the sidebar search–not having it is a deal-breaker in blog usability–is that Google’s small search box wanted to show the results right there in the sidebar. Like… um. Have you seen how long my post titles are? That was an unreadable mess. To add insult to injury, the full Search page wouldn’t read queries out of the querystring, so I couldn’t (by default) send requests there, either.

What I ended up using was this example of a two-page search, which allows me to put a no-JavaScript-needed search form in my sidebar. Searching there will take you to the search results page.

There’s nothing new in all of this, but it’s a constructed enough solution that it’s worth sharing. Two hours of time from top to bottom, which ain’t bad.

Related posts:

1. Search engines and me
2. Story engine
3. Lower-casing URIs in .NET

• How to Help Your Kids Build $25,000 Stock Portfolios – This is a tempting idea even with regards to something like weddings. "Help us retire together without having to eat pork-n-beans." …Naw, that doesn't work as well as with kids.
• 18 Five-Minute Decluttering Tips to Start Conquering Your Mess | zen habits – Since I haven't dedicated a full weekend to clearing out junk in a while, stumbling across this article was quite nice. Baby steps.
• Designing Microsites: Factors To Consider – Again, this is nicely relevant to the Character Sheet Manager project.
• Minimalism in Web Design: A Guide – I really like the tips here. I haven't started seriously approaching the (unauthenticated) public-facing design of the Character Sheet Manager, but since the site has such a singular focus, I'll be taking a minimalist approach to keep things clean and easy.

Related posts:

1. Tools, Consumption, and Sin-Eaters
2. “Pursuit” posted and character sheet news
3. Midnight linkage

“What brings you to my world, little bug?” the colorful woman asked Hardi and Robert. Her glance flitted between both of them before settling on Hardi. “Well?” she cooed, head tilting. Her accent was much lighter than Lucia’s.

Hardi straightened in her seat. “We’re looking for the people who have been attacking your world.” She paused briefly as six guards entered and took up positions in the corners and framing the door. “Are you hiding them here?”

Not much longer on Witches–just a few more chapters. I’ll probably spend some time with my lovely arties before starting another close-ended piece like Witches. I miss the more free-form style of the Transhuman Congress (as well as Lina and Harold, of course).

I had a pretty annoying blood donating experience this weekend. I had about 35-45 minutes in which to get the donation done, which is typically plenty of time for 09:00 on a Saturday. I ended up with a nervously chatty nurse, though, and was too nice to hurry her along. We talked about dreads and marrying white guys and biracial babies. Before I took my current gig, I could have said that it’s not the kind of discussion I had every day.

I have to admit, however, my politeness was stretched to its limit when she hit my vein wall, kept screwing with the tape, and had to rotate the needle for 10 seconds to relieve the pressure. Or when she scrubbed vigorously around the wound to try to clean off the iodine… which pulled at the hole in my arm. Or when the gauze got stuck (thanks to the iodine) to the hole in my arm, and she tried to pull it free in the wrong direction.

Momentary discomfort, luckily. If I’d also suffered from my usual light-headedness, I’d've probably been more… perturbed, but I preloaded with Sprite beforehand. Bleh, HFCS. Took an hour and ten minutes total, which was disappointing.

Any lingering discomfort was forgotten when one of the role-playing geeks (a former military field medic) showed me a video of an intraosseous infusion:

Um, yeah. Ow.

Related posts:

1. Faster, better, stronger.
2. “Planetfall” posted!
3. “The welcome goodbye” posted

• A Beginner’s Guide to Website Feedback – If I can wrap up and launch this damn character sheet app, stuff in this post will be handy for when it betas, especially the surveying. I suspect the LARPing audience will be sufficiently… opinionated to speak on it.
• Six Useful CSS3 Tools – Some of these are pretty slick, if you're moving into CSS3 development.
• Sharpening the blade, part MCMXVII: Nine Amazing Hours. – This is incredibly cool, and I plan to use it for a bit and see if it helps me focus.
• Amazing Examples of Paper Art – I almost hate to link to this, in case Greg gets ideas for elaborate projects.
• Python Business Rules Engine – Lott raises a good point about handling complex business rules, in that it's (often) cleaner and simpler to go ahead and incorporate complex business rules into the app itself rather than writing a parser to allow external entry. In my case, I have such a small user base on the side that would have been entering these rules that it's just as fine for me to do a small code release for any games added with these validation rules in them.
• YouTube – Turkish male belly dancer "diva" – Major glitter warning, here. Major. This may be the first male bellydancer I've ever seen who wasn't mocking dancing, and he's very good. I don't like the music or the dissolve and swirling transitions, though. Or the glitter. That's a lot of glitter.
• Amazon S3 and CloudFront with WordPress and DreamHost | .larre – This is quite a cool plugin. Not the quickest to set up with CloudFront, minifying, and combining, but worth the effort, even just for the hell of it.
• Girl quits her job on dry erase board, emails entire office – This is apparently fake, but a cute read anyway. I’d advise against airing dirty laundry like that, though.

Related posts:

1. Tools, Consumption, and Sin-Eaters
2. The noise of the night.
3. Rawr!

“But how do we get you out of here without anyone noticing?” Robert asked absently. Satisfied that her hand was fine, he pulled his hands back out of the attached gloves.

Hardi tamped down a grin of victory. “Nice and easy. You’re going to create a distraction, and we’re going to slip out.”

A small smile played on his lips as he asked, “What kind of distraction?”

No links this week. I’ve been planning hard on the Geist Character Sheet Manager, working with Greg to come up with and prioritize requirements. I’m playing around with the Agile “story” idea. It seems like a good informal way to gather reqs, though I’ve done most of the generation of the ideas. Greg should really be doing the bulk of them, since he’s a prime example of a user on both the game manager and character sheet user sides. He’s been great about helping come up with cases that fit World of Darkness games I don’t play and giving specific examples of things I have to account for.

I’ve got a preliminary feature set for my first iteration of the game manager side, and boy, did it break my proposed timeline. Depending on what my priority cut-off is, it may be late September to mid-October before it’s ready as an alpha. Some things may turn out to be simpler to implement than I’m currently estimate. For instance, either in this iteration or the next, I’ll need to write a rules system to allow the game manager to put in everything from prerequisites to conditional dot cost changes. I’m looking forward to designing and implementing that.

And so I keep moving.

Related posts:

1. Weekend linkage
2. Tools, Consumption, and Sin-Eaters
3. Tidying up LARP character sheets: an app in progress

I’m never sure how robust to build out something like this. It’s going to prove handy to me in the case of this app and others I’ll be building out, but there’s no way I’m going to try to implement every nook and cranny of Redmine’s API — I don’t need it. If I release it, though, other people could want the things I don’t need. What’s the right balance in releasing a tool?

As always when working on an app, it’s a struggle not to over-engineer things. This Character Sheet Manager project consists of two apps (ignoring the Redmine integration): the Game Manager and the Sheet Manager. The Game Manager is (right now) just a data app — it’s for me (and Greg) to put in Merits, Skills, various powers, etc. The Sheet Manager will be the consumer of said data.

But character creation logic — knowing how many skills someone gets, knowing how many specialties — is really per-game knowledge. But it’s the same for all White Wolf games of this version. …But what if I want to support something like Scion, where the rules differ a bit?

But… I don’t want to support them right now, and might not ever. Is this also a case of You Ain’t Gonna Need It? Right now, I kinda just want to get it out the door and in use as an evolving prototype. I don’t mind refactoring and rewriting some, but I don’t want to shoot my foot completely off, either.

In related news, there’s apparently been grumbling in global-level out of character Geist mailing lists about the existing character sheet. I should hurry.

Related posts:

1. Weekend linkage
2. “Pursuit” posted and character sheet news
3. Tidying up LARP character sheets: an app in progress

• glenscott.net » Restore an Eee PC 701 back to factory Xandros from a USB stick with no ASUS Support DVD – This proved quite handy last night in restoring ol' Tammy to defaults for Greg.
• The legend of the superprogrammer – "Caper Jones, in an unpublished 1977 study for IBM, found that the very best developers are much more productive than the worst programmer — when working on small projects. The best developer will complete a 1k line of code (LOC) effort 6 times faster than the lousiest. The productivity delta falls to 2x on a 64k LOC project. Beyond a few hundred thousand LOC both sorts of people perform equally well. Or equally poorly."
• Skepticblog » The Reasonableness of Weird Things – "Many people have quite good reasons for believing in the paranormal. [...] In my experience, the top reasons people believe weird things are not only understandable, but identical to the reasons most skeptics believe things: they are persuaded by personal experiences (or by the experiences of a loved one); or, they are persuaded by the sources they have consulted."

  Thank you, thank you, thank you. I get so sick of elitist, skeptic douchebags with their holier-than-thou lock on critical thinking that I — despite having the same desires to protect people against fraudulent untruths — won't put the label of "skeptic" on myself. It's a movement that has a rude, pushy face in far too many of the cases I've seen. I hope more folks take articles like this to heart.

• Think Progress » Racist New Hampshire State House Candidate Advises Tea Party To Be More Open With Its Racism – "For far too long white Americans have been told that diversity is something beneficial to their existence. Statistics prove that the opposite is true."

  Um. Okay.

• Because Blacks are not American | Prometheus 6 – Too bad I didn't teach a "fuzzy" subject like history. Then I could have had the opportunity to really screw with people's heads.
• Buttersafe » The Essence of Being Human – Mmm, cat videos.
• 10 New High-Quality Fonts for Your Designs | Freebies – I always love things like this, even if — for the web — it means slicing and dicing text as images. To that end, if sites could consistently and easily push fonts out to users, would that be a good thing?
• Is It Time To Quit Your Day Job? – One of those nice "get off your butt and do your passion job" articles.
• High Test Coverage Ratio is a good thing, Anyway! – Patrick Smacchia [MVP C#] – CodeBetter.Com – Stuff you need to Code Better! – Patrick uses the broken window argument to conclude that 100% test coverage from the start will encourage developers to maintain 100% coverage in code as they refactor. I'm not sure I buy that for cases where rapidly-developed prototypes are refactored and developed into deployed applications.
• 70 Unique Examples Of 404 Error Pages For Your Inspiration – These are gorgeous and helpful. And inspiring.
• How the Old Spice Videos Are Being Made – Don't know about you, but I can't get enough of Mustafa.
• Rules For Games: Do & Don’t #1 | Rock, Paper, Shotgun – "Don’t: leave diary entries by one person scattered over miles of corridors, buildings and countries. That’s not how a diary works."

  …And more.

• The Dawn Chapel – Firefox has crashed – I like cookies. And cuteness.
• Siobhan – “they were going to strip search me … this feeling, this experience, is the hardest for me to reconcile and one of the times in my life I have felt the most powerless” | G20Inquiry.org – I've evidently been sleeping on some serious shit going on.
• Arab guilty of rape after consensual sex with Jew | World news | The Guardian – Leaving aside the crazy Israeli politics and the legal side — is deliberately lying about something to get sex with someone rape if you know the thing you're lying about would make the other person say "no"? As of this moment, I think so.

Related posts:

1. Tastee (del.icio.us) links! (December 9th)
2. Tastee (del.icio.us) links! (April 21st)
3. Tastee (del.icio.us) links! (December 2nd)

“Hardi, why in all the worlds wouldn’t we tell Wilder about this?” Robert asked, waving a hand at her comm. He’d just listened to the ransom note for the second time, and was sitting at the desk with his feet propped up.

“Yet. Not tell him yet.” Hardi stood still in the middle of the room, staring at the floor with her hands tucked in her pockets. “And I… don’t know,” she said slowly.

After that deep voice had threatened to go on “vacation” for a week, Hardi had listened to the message again. And again, until the rough voice was etched into her memory. Robert had walked in on her standing as she was now, letting the file loop. She’d looked at him guiltily, but let it play for him.

Robert sat leaned back and balanced his chair on two legs. “Have you had one of your visions? Does it feature you as the heroine, saving the planet from the bad guys?”

This has been a far crazier week than I thought it would be, hence the lack of writing here in any form. I worked at the hospital last weekend, and as I was moving an absolutely insane number of charts–more charts than I’d seen in about two years, when we starting doing a lot more digitally–I reminded myself why I have that job. Multiple streams of income. I’ve Greg started a new D & D game this week at the house, and I’m getting to play with all-new folks. A couple of guys from the LARP I do, one guy from work, and good ol’ Nathan.

Greg tried something this game that we’d never done before, and that was to use the Dawn of World (PDF) rules to have us collectively generate a world. We played D & D gods–Avandra, Zehir, Ioun, etc.–and created the world with our own flavor, everything from mountains and rivers to D & D races. Greg let us go decently far with changing stereotypes, so we were able to build civilizations with specializations and tech that differs from the stuff in the books. We’ve got a floating island, a nearly wiped out Elandrin civilization, a protected civilization of Wilden living in and around a forest of… well, let’s say Melora’s avatar is named after THC.

He’ll write up the world history from that session and make a world map, and then next session we’ll make characters in that world and be off on our crazy adventure. I think it’ll be a blast, especially once all the players learn about each other and settle into things.

Related posts:

1. “Breached defenses” posted
2. “Planetfall” posted!
3. “Pursuit” posted and character sheet news

There are going to be spoilers here, but nothing that actually ruins the novelty of the book, I hope.

What resonated with me were the times when Dan Freeman (the main character) questioned himself, because he was propped up as a bit too perfect for the role of lead revolutionary. As he left the CIA, he wondered, “Have I given in to this middle-class thing?” He marveled at how easy it would be to be his cover identity: to have the women, the “respect” of white people, the nice apartment and the money. To be a whole person, though, he had to maintain two lives.

There are so many revealing gems, great uses of language and parallelism, and instances of pushing beyond the comfort zone. Freeman asks several times in the book, “So, how’s that working for you?” about the well-behaved “just keep waiting for them to notice that we’re worthy” movement. The answers were, of course, desperate and avoidant. The idea that it’s that latent, suppressed fear that would cause an uprising, not just the frustration and anger, was carefully built up throughout the book. The idea of education having value in and of itself, aside from its worth in pursuing a career.

It was such a different time… and yet some things still shine through. Problem is, this book could incite someone like me to become an apologist. “I feel bad for being a product of the privileges of integration and good education and blending that’s gotten me where I am.”

I’m going to have to hunt down the movie now; the library system here doesn’t have it.

I’ve gotten very little app development done these last two weeks. I’m still about 16 hours from a working prototype of the character sheet system for Greg to beat up, but I’ve been relaxing a little more in my spare time. Torchlight is fun. So is Left 4 Dead 2. Spook was a great read. So was Not Becoming My Mother.

As I’ve been figuring out how people work on this retirement saving thing (speaking of being middle-class), two questions have come to mind:

1. If you can only contribute $5k a year to an IRA, how do you handle wanting to invest in 5-7 different mutual funds that each have $1000-3000 initial investments? Dump money into a taxable account and split your portfolio? Leverage your 401k more (even without employer matching), which doesn’t seem to enforce investment minimums?
2. If you use a discount broker like Sharebuilder, how do you handle 401k rollovers? I asked if I could keep the securities (investments) rather than having to cash out, and they said, “only for those securities Sharebuilder carries.” Which are, like, none. That’s partly why I’m using the 401k — to diversify more. So would I open a rollover account at a brick-and-mortar bank for the rollover? Or at some fund family site, like Vanguard or T. Rowe Price? I hardly want more bank accounts.

That brings me to another thing: asking for advice. I read Get Rich Slowly (awesome blog, by the way), and occasionally, J.D. hits on the idea of the “millionaire next door”. That person who lives frugally and drives the 10+ year old car and eats at home. (I’m happy to say that I know that guy. ^_^)

For those of us late to the game–and maybe I only feel so late in comparison to him–I think one of the most valuable things the millionaire next door can do is share their knowledge. Not lecturing about not eating out or clipping coupons or frowning on big-ticket purchases. I’ve found that frugality–like weight loss–is something internal, with the external aspects being the sharing of processes, not validation. So not frugality, but answering questions like the two above and:

• How and where did you start saving?
• What were your sources of info? (Esp. if it wasn’t parental.)
• What do you consider reasonable risk in terms of investing?
• Do you even invest? (I know a few people who feel the speculation of stock market isn’t something they want to be involved in.)

I get the feeling that people are wary of seeming like they’re passing on financial gospel, like we want the best stock picks, a formula for success, or the ultimate portfolio. I’d love to sit down with someone with financial experience who isn’t getting paid to sell me a product and just shoot the shit about financial organization and knowledge.

Most of the people my age are as new to game as me, and those who are older are much more risk averse or don’t geek about this. Money goes into the 401k and they don’t worry about it except to freak when the stock market plummets.

Hmm. I’ve been just soaking in knowledge and learning lately. Life is good.

Related posts:

1. Mo’ money, mo’ problems: a retirement app idea
2. Take a vote
3. John Grisham’s The Firm

• louisgray.com: Why I Turned In My iPhone and Went Android – I love this article. I've written before that I'm considering switching, and at this point, it's only the cost of breaking contract that's holding me back.
• YouTube – heavymedic dance work in progress – unfinished – Y'all know I'm a doc/heavy shipper.
• Penny Arcade – The Sound Of A Klaxon – Regarding the iPhone 4: "She sees that phone the same way she would see a hat, or a ladle. If you had a ladle and you tried to get soup with the ladle but the soup came out, or fell out, or whatever the culinary equivalent of a lost call is, she would try to break it over her knee before throwing it in the garbage."
• Prince Declares Internet Over – Yes, that Prince. The Artist Formerly Known As might actually become "formerly known" among the next generation with his mysterious disappearance from the Internet. Strangely, I hear his music on Pandora. Maybe that won't be for much longer. "Unlike most other rock stars, he has banned YouTube and iTunes from using any of his music and has even closed down his own official website."
• Narrower Towers: 20 Of Japan’s Thinnest Buildings | Design + Ideas on WU – From Greg, this is a really cool look at very slender architecture — 7-story buildings with 1,200 sq. ft footprints (feetprint?), a building that looks less than 10-feet wide, but supposedly has a Chinese restaurant on the ground floor, etc.
• The Lazy Way to Investment Success – I'm not sure I like the term "lazy" for this style — you still need to do your due diligence on picking the index funds or ETFs, but I like the simplicity of the style.
• Buttersafe – Updated Tuesdays and Thursdays » Archive » 11 Bad Comics – Clowns and frowns.
• 2 officers out of jobs in wake of repeated Tasering of woman  | ajc.com – Um. Read this.

Related posts:

1. Quasi-daily linkage
2. Quasi-daily linkage
3. Is Soulja Boy Ze Frank inspired?