Gabriel Weinberg's Blog

How-to stop most people from spidering your site and stealing content

2009-09-15T17:35:15Z

I run a few sites with a lot of content that I don't want spidered by anyone other than the major search engines. At best, undesired spidering eats at your bandwidth and page response time. At worst, it can lead to widespread stealing and duplication of your content.

Anyway, this post details exactly what I currently do to prevent such spidering. All the code is in my new git repository I created to share code with you.

I created a DB named logip, owned by user logip, and then added these tables. The logip table records requests from IP addresses I am tracking. And the badips and badips2 tables hold IP addresses I am presently blocking.
I set up this Perl script (logip.pl) to run all the time. It tails my Web server log file and adds IP addresses to the logip table. There are variables at the top you can use to exclude certain virtual hosts, e.g. a site with few pages where you aren't concerned with crawling. You can also exempt certain IPs, e.g. your own.

logip.pl currently only adds IPs for requests with 200 HTTP status codes (OKs). It also skips images, css, js, etc., IP addresses known to be associated with the major search engines, and repeated requests for the same page by the same IP (hard refreshes). The idea is to record unique successful requests of actual distinct pages from non-search engines.

I run the script via daemontools, but you could run it through inetd or whatever. If daemontools interests you, the commands I ran to setup management are in logip.sh. If you use those commands, you will want to change them to point to your svscan and logip.pl script directories appropriately.
I set up this Perl script (badips.pl) to run periodically. In particular, I have it set up to run every minute via crontab. The frequency at which it is run is the minimum frequency that new violators of my spidering policy will be blocked. So if you run it every minute, people will have (on average) half a minute to grab your stuff before you start blocking them. I haven't found in practice an urge to make the time interval smaller, but if a lot of people desire it, I could rewrite the script for that purpose.

badips.pl works on a threshold basis. It looks at various tunable timeframes and checks whether new IP addresses have exceeded a page request threshold for those timeframes. The ones I currently use are in the script. For example, 20 page requests in the past minute, or 50 over a day. You can tune these to what is appropriate for your sites.

The second variable is whether to log violators in either the badips or badips2 tables. The distinction is whether you think a violation is really bad or just pretty bad. For example, I currently mark passing the minute threshold as really bad and all else as just pretty bad. A pretty bad block stays around for 10 days, whereas a really bad block stays around for 180 days.
The output of badips.pl is a configuration file that nginx or Apache reads on the fly. It works with both Web servers, and there is a variable at the top of the file to indicate which one you are using. The resulting conf file is a bunch of Deny IP lines that lists out the current IPs you are blocking.

For Apache, there are a some other lines that preemptively block suspicious user agents, e.g. curl and wget. I haven't yet ported these preemptive lines over to nginx. The intention is for Apache to see the file via an Allow Override All directive, i.e. via a changing .htaccess file. For nginx, the configuration is reloaded on the fly.
If new IPs are added, you are sent an email notifying you of the new block(s). The script attempts to do reverse DNS on the IP and the forward DNS on that host to give you some context. For example, if it is a Google IP, you will want to unblock it. However, in practice, I haven't done that in a while because those IPs are well exempted in logip.pl. badips.pl also cleans up the DB at the end of the script before exiting, deleting expired records and vacuuming the table (for PostgreSQL).

I've evolved this process over the last few years, and it works quite well for me and my sites. Your feedback if of course welcome. I'm always looking for improvements and am willing to make them.

I am aware that the current process has some holes. The two biggest are:

You can spider successfully via a large number of IPs, most notably the TOR network. In the past, I have added those IPs dynamically, and I might do that again. Adding a ton of IPs slows down the Web server considerably, however. This is why I backed off of that approach in the past.
You can grab pages really slowly. That is, if you stay under the thresholds, you won't get caught by this system.

Enjoy!

Rand Fishkin of SEOMoz.org on Traction

2009-08-31T18:39:11Z

This is the second episode of Traction, an interview series with successful entrepreneurs that demystifies how startup companies get traction for their businesses. Getting traction is the process of acquiring users and customers, and more generally significantly growing businesses.

In this interview Rand Fishkin tells the story of SEOMoz.org, a leader in search engine optimization (SEO) software tools. Rand explains how he got traction organically, through tireless blogging, picking the right market, and of course SEO. The interview is ~30min.

As you may or may not be able to tell, I had some major video issues again. I had to re-sync the audio to the video, and my whole video stream was corrupted. It might actually work better without me in the picture--let me know what you think.

Traction with Steve Barsh

2009-08-04T15:08:00Z

Today I'm launching Traction, an interview series with successful entrepreneurs that demystifies how startup companies get traction for their businesses. Getting traction is the process of acquiring users and customers, and more generally significantly growing businesses. I chose this subject because it is the area I see entrepreneurs struggle with most.

In this inaugural interview, Steve Barsh tells us about two of his ventures: SECA, a bootstrapped software consulting turned product company he started out of college and later sold to MCI, and iGrandparents.com, a Web 1.0 VC-backed company he took from an idea on paper to capturing 1.5% of the baby-boomer market.

Steve shares some of his tips, tricks and traps including how he "leans forward" into his markets, how he effectively uses PR and how getting traction relates to positioning. Steve is currently a partner at Dreamit Ventures, a pre-seed investment firm in Philadelphia. This video is about 30 minutes long.

This episode of Traction is hosted by blip.tv, where you can also get the mp3 (audio only). To get new episodes, subscribe via iTunes or RSS. I will also be posting them on my blog and twitter.

The technical quality of the videos should improve over time, subject to the limitations of Skype. I had some issues with this first interview, namely the video and audio tracks becoming way out of sync. I attempted to re-sync them by hand until I ran into major diminishing returns.

If you have interviewee (or other) suggestions please let me know. I would greatly appreciate any feedback. This interview is perhaps a bit too long. However, Steve is talking about 2 separate businesses, so it is really two interviews in one.

Data Kitteh now tells you what sites are behind IP addresses, for free

2009-07-30T20:27:33Z

Data Kitteh is a new site I put up to house my personal data visualization projects. Today I'm adding a third project to the site: a free listing of what sites are behind IP addresses.

I find myself going to Domain Tools for this data all the time. But I never want to pay them anything, and they only give you a few sites for free. They do give you server location info for free, which I am also always looking up, but it sits on another page in their site.

So what I did is I took a byproduct of the Parked Domains Project, i.e. IP address <-> domain name linkages, and I sorted by IP address. I also added more value by including server location as well as whether the underlying domains are parked, non-parked, or don't have a live site behind them.

I did put up a cutoff of 100 domains per IP address because some domains have literally tens of thousands. However, this is a vast improvement from the 3 you get at Domain Tools and has proven to be more than enough for my purposes.

I have coverage right now for these TLDs: .com, .biz, .org, .us, .info, .coop, .aero, mobi and .net. This gets you most domains, but obviously the vast majority of TLDs (mostly countries) are not covered. The problem is I don't have good domain lists for these TLDs. If you do, e.g. for .uk, .au, etc., and you would like to share them with me, I would greatly appreciate it. Also, if you have any feature requests, I will surely consider them.

Health care policy: will we ration 5% cures?

2009-07-30T18:21:21Z

Suppose you are terminally ill with few months to live if untreated. There is a new drug that has a 5% success rate that if successful, will leave you completely cured. But 95% of the time it won't work and you will get no benefit. Should the government pay for these 5% cures?

On the face of it, this is a case where pure cost/benefit analysis would most often return a big fat NO. This result occurs because 95% of the costs will be effectively wasted such that cost per unit of life extension will seem really high relative to other things under consideration. It gets even worse if the cure ends up being say a 3-year life extension on average because those patients end up dying from something else in that time frame due to their weakened initial state. Or if the cure rate was even lower, say 3% or even 1%.

At the same time, the cost to the person whose life is saved appears ~20x lower to them because they would not incur individually the costs for the other 19 people (on average). I don't pretend to have the answer to the question, and would be interested in your thoughts, but I think this is an interesting case that shows the complexities we will soon increasingly face as well as one potential pitfall of cost/benefit analysis that pundits love to push.

We ration health care already, both privately and publicly, so I'm not too interested in the question of will we ration care of not. I believe that debate to be a false one. What I am interested in, however, is how our rationing will change under proposed reforms. It seems that we are headed towards some sort of quasi-public body that will decide, at least for public options, what care will be approved for varying reimbursements. Of course, this is similar to what we do now, but this new body will presumably have more power and be trying a lot harder than we do now to squeeze costs out of the system.

No one really cares about unnecessary costs because they are by definition unnecessary, e.g. duplicate tests. And I suspect existing Medicare benefits will end up being frozen in time, i.e. no current benefits will be cut. Yet that is only a short term win since new drugs enter the market place all the time. It is around these new drugs where some interesting debate will occur, which brings me back to the case of what I'm calling the 5% cure.

These cures pose additional problems beyond the initial cost/benefit debates. Suppose we stop covering 5% cures as a rule. Further suppose, and I don't know if this is the case but I suspect such, that a lot of great treatments that end up being 50% cures start out as 5% cures. By shutting down (or at least hampering) the 5% cure market we will be effectively telling pharma not to invest in these cutting edge treatments. From their perspective, what's the point? That could lead to less important disruptive innovation in health care, which is the very thing we want to encourage.

Nobody puts baby in the corner

2009-07-29T11:25:22Z

Eli's 4 1/2 months already(!) and he crawls. Not quickly mind you, but he eventually gets where he is going, sort of like a snail. Butt high in the air, slinking along, and not terribly happy about the process as much as the destination.

Our doctor told us that babies are smarter than you think they are. I thought I was heeding his advice perfectly until last night. I had just assumed (key mistake) that his attempts at crawling were in somewhat random directions.

We had been putting him to bed in a crib at night, and he would make his way over the course of the night into the corner. It looked uncomfortable with the head pressed up against the side, so we would move him back to the center. But he would find he kept finding his way back to the corner.

Yesterday when putting him in the crib for a nap, I decided to experiment and just start him in the corner. He went to sleep very quickly.

OK, but maybe that is because he had nowhere to go. So last night we tried again to start him in the corner. He has taken to going to sleep between 6:30-7:30 and waking up around 12:30-1:30 to cry/"crankle" for a while (30-45min) only to fall back asleep for several more hours.

He woke up last night around 1:10AM. I go in there and he is along the top edge of the crib, faced in the other direction, butt up, about 1/3 of the way to the other corner. It really looked like he was making his way from one corner to the other, deliberately.

So I picked him up and put him where he seemed like he wanted to go. And it worked!

Time commitment for being a great dad? Great mom?

2009-07-28T22:45:59Z

My last post taking issue with Obama's fatherhood commercial generated some passionate comments summarized by "I think the message he's trying to convey is that being a [great] father is not that hard or time consuming."

This got me thinking as to what is the minimum amount time a parent needs to spend with their child daily (on average) to be even considered great? Can you skate by on just a few minutes, if they are somehow quality minutes?

Personally, I don't think so. Certainly it takes more than just a few minutes, right? What about an hour a day? Can you still be a great dad on an hour a day?

I'm still hesitant to call that kind of parenting *great* as in exceptional or distinguished. At best it seems good, and is probably more like average.

And is the requirement for being a great mom different than a great dad? If the traditional working family was reversed with the mom working and the dad staying at home, and the mom spent one quality hour a day with the kids, would she be a great mom?

She better be or its sexist, right? Yet I can't see that person thinking she is a great Mom.

What if both parents work and they each just spend one quality hour with their kids, paying childcare workers for the rest of the time. Can they both be great parents through just this one shared hour a day?

Don't get me wrong. I agree with the message that you should spend more time with your kids, and all time counts. I think the problem is in making the leap from "a few minutes" or even an hour to somehow being "great."

Obama "Fatherhood" commercial sends terrible messages

2009-07-28T12:43:35Z

I hate this Obama "Fatherhood" commercial (15 sec) I keep seeing on Hulu. Here's the transcript:

"To be a great dad is the most important job in a man's life, but it doesn't have to be hard. All it takes is a few minutes of your time. Because the smallest moments can have the biggest impact on a child's life. Take time to be a dad today."

So let me get this straight. You can be a great dad through just a few minutes of your time? On the contrary, a few minutes is the difference between an absolutely absent dad and a severely neglectful dad, but it doesn't come close to being a great dad or even an OK dad.

Also, I gather from this ad that parenting doesn't have to be hard because all it takes is a few minutes of your time. Really? So who takes care of the children the other 3,557 minutes of the day? Is he really implying that the mother's role is to do so?

It also says being a great dad is the most important job in a man's life. If it is that important, doesn't it require more of us than a few minutes?

Banned words in Google's SafeSearch

2009-07-27T18:56:43Z

I couldn't find a definitive list of banned words in Google's SafeSearch w/ strict filtering (this site is the closest I got), so I set out to make one myself. Here are the results:

analsex
anus
asshole
babes
bastard
beastiality
bdsm
bitch
bisex
blowjob
blowjobs
bondage
boob
boobs
bullshit
bukkake
clit
clitoris
cock
cocksucker
cum
cumshot
cumshots
cunt
dildo
dildos
erotic
erotica
erotico
fellatio
fetish
fisting
fuck
fucked
fucker
fucking
gangbang
groupsex
hardcore
hentai
horny
kinky
mosthfuckers
motherfucker
naked
nazi
nude
nudes
oralsex
orgy
pervert
porn
pussy
shemale
shit
sluts
softcore
teensex
tits
twat
upskirt
upskirts
xxx

I'm very interested in getting a comprehensive list. Can you find anything else to add?

Obama, please tell Congress to read their bills

2009-07-26T23:41:07Z

Obama, please say you won't sign any bills that Congress hasn't had the opprortunity to read, digest and debate.

I realize you are not in charge of Congress. But you are the head of the party in power and there is a lot more you could be doing to "change Washington" like you said you were going to do. Sending this signal would make a great start.

Maybe no one in Congress is to blame. Maybe the root cause is the system itself, which drives the tactics that result in last minute, hastily written and unread bills. If it is a systemic problem, that is more of a reason for you to exert influence because by definition it wouldn't change otherwise.

So at your next press conference, please make the point you will expect more of Congress here on out.

Earliest still active (and unprotected) twitter users

2009-07-24T21:07:53Z

I've been messing around lately with twitter network analysis and follow recommendations for my own amusement. While using the Twitter API I randomly came across a lot of really low user ids, first because I follow @joshk, and then because a lot of other people follow @ev & co.

I guess twitter more or less assigned user ids sequentially from 0, at least early on. So I went back and queried the earliest user ids. The following is a list of the ones under 1K that are still active (and unprotected). For the fun of it, I also did an analysis of each with follow recommendations.

12 @jack analysis
13 @biz analysis
15 @crystal analysis
17 @tonystubblebine analysis
18 @Adam analysis
20 @ev analysis
21 @dom analysis
22 @rabble analysis
23 @timroberts analysis
31 @rayreadyray analysis
34 @ariel analysis
47 @kellan analysis
52 @hook analysis
53 @sara analysis
56 @kati analysis
57 @SarahM analysis
58 @Darkside analysis
59 @Tim535353 analysis
60 @andrew analysis
61 @dan analysis
62 @gregkidd analysis
64 @livia analysis
66 @dunstan analysis
67 @andykeep analysis
87 @krissy analysis
89 @al analysis
94 @alissa analysis
104 @RobMcCarthy analysis
107 @pud analysis
108 @ilona analysis
126 @brandooon analysis
136 @meikel analysis
150 @stevej analysis
155 @craigcalef analysis
187 @stacy analysis
189 @maestorm analysis
190 @jokerwonga analysis
194 @jennycool analysis
202 @russ analysis
203 @smays analysis
204 @texasex2k analysis
209 @phil analysis
221 @h0mee analysis
224 @davepell analysis
242 @victor analysis
246 @blaine analysis
257 @rael analysis
259 @ian analysis
260 @sharon analysis
266 @japhy analysis
274 @danarkind analysis
277 @casey analysis
278 @dreamdawl19 analysis
279 @Angela analysis
288 @solsken analysis
291 @goldman analysis
292 @cw analysis
293 @mai analysis
294 @ario analysis
295 @joshk analysis
314 @ethan analysis
322 @twang analysis
334 @eggplantia5 analysis
341 @alexia analysis
344 @Bob analysis
346 @jo analysis
350 @oof analysis
357 @wubbahed analysis
364 @graysky analysis
365 @Gary analysis
380 @kevin analysis
388 @aron analysis
391 @TylerDurden analysis
398 @Quinn analysis
407 @asa analysis
408 @endquote analysis
409 @Case analysis
414 @veen analysis
418 @dens analysis
422 @shellen analysis
423 @SaraWD analysis
424 @katalia analysis
430 @Richard analysis
432 @cb analysis
443 @atvelu analysis
448 @Maggie analysis
450 @LoriLoo analysis
456 @davegray analysis
460 @andr01d analysis
506 @JonMarkwell analysis
509 @jerrymichalski analysis
521 @gerwitz analysis
528 @buzz analysis
531 @DocRob analysis
535 @bakingfairy analysis
541 @lane analysis
544 @peter analysis
556 @ch analysis
562 @rubylabs analysis
573 @heyitsnoah analysis
577 @Nica analysis
581 @Ari analysis
586 @sacca analysis
590 @Derrick analysis
591 @Naina analysis
598 @sfegette analysis
607 @mg analysis
613 @jerry analysis
614 @dcharrison analysis
633 @zephoria analysis
637 @Dick analysis
642 @phopkins analysis
648 @heif analysis
649 @MaryHodder analysis
652 @kohtz analysis
666 @Jessie analysis
675 @Anthony analysis
681 @Yarrow analysis
683 @jcherup analysis
684 @kushwaha analysis
686 @ericost analysis
690 @avartan analysis
711 @Alysha analysis
714 @dereko analysis
717 @Manu analysis
718 @jshugars analysis
719 @m2mcguire analysis
722 @neha analysis
728 @codefin analysis
730 @crutis analysis
746 @gshellen analysis
747 @thepartycow analysis
750 @cg analysis
753 @mc analysis
754 @superc analysis
755 @noaml analysis
760 @boogah analysis
761 @rodbegbie analysis
762 @astroboy analysis
763 @Eddie94603 analysis
765 @seanbonner analysis
767 @xenijardin analysis
769 @RichardAult analysis
775 @ravivarmapm analysis
780 @willij analysis
781 @may analysis
785 @kfury analysis
791 @paulmorriss analysis
795 @Wrickr analysis
841 @colinschlueter analysis
842 @brs analysis
844 @brough analysis
845 @damien analysis
849 @largeheartedboy analysis
850 @eventi analysis
852 @emorris_google analysis
855 @ikisai analysis
859 @richziade analysis
861 @jaygoldman analysis
866 @SEKeener analysis
874 @waynesutton analysis
875 @aaronbailey analysis
878 @everett analysis
880 @nanek analysis
882 @nerdgirl analysis
885 @Ross analysis
888 @caroline analysis
892 @laluna_negra analysis
897 @manton analysis
903 @Hunter analysis
911 @richellis analysis
913 @alba analysis
916 @mojodenbow analysis
917 @JoeLaz analysis
918 @mac analysis
922 @Holger analysis
923 @earnestdotcom analysis
926 @bradbarrish analysis
927 @erindecker analysis
928 @necrodome analysis
934 @thatsdope analysis
936 @karina analysis
939 @michael analysis
940 @jrhyley analysis
947 @leeott analysis
949 @pedraum analysis
951 @axodys analysis
953 @mrdonut analysis
955 @claysmith analysis
958 @tyler analysis
959 @fix analysis
960 @velkr0 analysis
961 @FakeRiaz analysis
963 @scootrous analysis
964 @getluky analysis
965 @cvodb analysis
977 @beaulebens analysis
978 @ddukes analysis
981 @Morgan1st analysis
984 @brentv analysis
985 @mastermaq analysis
987 @lorna analysis
988 @nitin analysis
989 @om analysis
992 @heyalchang analysis
993 @joshua analysis
994 @kareem analysis
997 @tedr analysis
999 @dariusmc analysis
1000 @jordym analysis

P.S. If you want an analysis, follow @datakitteh and I'll get to eventually.

I love TV! There, I said it...

2009-07-16T14:21:41Z

Woke up today to see yet another anti-TV post on Hacker News. The comments link to two really funny takes on the issue at Stuff White People Like and The Onion.

Here's what I like to watch on TV:

Educational

Comedy

Drama

Sports

Tour de France
Tennis
Soccer

I find it odd to indict an entire communication medium with anything. Of course there is stuff you are not going to like on TV. But by the same token, there is probably stuff you will like too.

Do you want to be good friends with the average person?

I can understand the argument of wanting to reclaim your time. I cut out reading the newspaper in paper-form last year. I used to get the WSJ and the Phoenix delivered. But I didn't cut out the entire news medium. Instead, I just cut down my consumption to what I find most interesting, which I now get through a site I made for that purpose and via RSS feeds.

Similarly, cutting out all TV seems a bit drastic me. Ne qid nimis (nothing in excess). I DVR everything, so I watch it when I want and I get to skip all the commercials.

Anyway, my point here is not really to defend TV. I'm just saying I consider myself a hacker and I, for one, love the above shows.

What do you like to watch?

Global innovation loss from lack of co-opting startup experience?

2009-07-15T12:54:09Z

How often have you seen a good product die? If only they had done x, y or z... Perhaps if they had a team with more or varied experience we might all be better off for their product's continued development.

Now sum up the innovation loss from that one product to all the products that have died for lack of execution. How much global productivity have we lost? As high as 1%/yr? More?

I'm applying this thought experiment to startups, but the same question could be applied to anything where experience plays a role. The topic occurred to me while watching Robert McNamara on Charlie Rose last night on the subject of military strategy (rebroadcasts).

McNamara has all these lessons that he has written up in several books and that were depicted in the movie The Fog of War. In the startup world, we see lesson posts coming through Hacker News daily. Despite the authors' (and often the readers') best intentions, I'm highly doubtful that this attempt at knowledge transfer is having much of an impact.

Certainly, writing up lessons is better than not writing up lessons. And certainly the Internet has helped spread experience faster and more democraticaly, e.g. Stack Overflow. Yet there is only so much you can get across in a hands-off approach.

I think co-opting hands-on experience is the key value of the pre-seed funds like YC and DreamIt. I am a mentor for Dreamit participant Scribnia this summer, and so I've seen this first hand. It is the difference of reading my post on SEO and having me review your specific site in detail.

Don't get me wrong, Ask HN is a great start. I'm just not convinced it is enough to stop the massive productivity loss I think is occurring due to lack of experience. I don't have (or pretend to have) the answers, but here's a couple ideas that popped into my head:

Startups could more intensely focus on the smart money when raising capital. They are getting people literally financially invested in their companies. If those people have the right experience, the startup has (I hope) motivated them to apply that experience to the company.
Similarly, startups could co-opt experience by creating other financial incentives. For example, an advisory board where you give each of a few advisors 1%.

What else?

What makes you weird?

2009-07-06T12:13:00Z

What does most (>75%) of the population (of your country) do that you do not?

Note that I'm not asking what makes you unique. For example, I'm an INTJ, which is one of those 1% personality types. However, it's not like 99% of people are one personality type and then us INTJs are the other one. There are 16 Myers-Briggs types, so this doesn't count, although it certainly contributes to my uniqueness.

I'm looking for (at least almost) binary things. The first thing I thought of is that don't drink, but it turns out 40% of Americans don't drink. So that doesn't count.

Here's some things that seem to count (with my % estimates):

I have a graduate degree (10% of US pop>25). (A bachelor's degree doesn't cut it any more as 27% of the US population over 25 has one.)
I went to a top-10 school (3% of school goers).
I am a business owner (15% of US pop>25).
I did not get a job after school. (I started a business instead.) I can't find a stat here, but I'm sure it is very low.
I'm a stay-at-home Dad (1% of dads).
I blog (12% of Internet users).

What about you?

Things about Web Images I Just Learned

2009-06-16T14:17:23Z

I thought I knew everything you needed to know about Web images. But, of course, I didn't. Here's what I just learned when launching the new icon bar on the homepage of Duck Duck Go. We wanted the it to function sort of like the Apple dashboard (and on the Web like Schmedley's bottom bar).

img{-ms-interpolation-mode:bicubic}. Short version: if you resize images dynamically, they will look bad on IE unless you put this in your CSS.

Longer version: We ended up using the YUI Animation Library to do the animation. But no matter how we did it using 1 image, it always looked terrible on IE. Even if we used an image exactly as big as the big size, and did the smaller image exactly half of the bigger size (which should be easy to resize), it still looked bad.

So then we tried using two images, which sort-of worked, but had its own issues. Sometimes it would slow down the animation. It used almost double the image size and requests (a big no-no), and the actual resizing still looked bad (as opposed to the endpoints)!

This was unacceptable, so I decided to dig deeper on the Web about this issue. It turns out modern browsers use Bicubic interpolation to resize images and make them look good in the process. For whatever reason, IE7+ has decided to turn it off by default. I'm guessing this is because it takes some processing power, but it renders resized images looking terrible so I personally don't think this is a good trade off. Anyway, if you add that above CSS to your page, IE7+ will use this method and your images will look good. I suppose I never hit this before because usually you shouldn't be resizing images dynamically. But there are cases where you want to do it...

Unfortunately, it still doesn't work for IE6, on which you need to use the good ol' AlphaImageLoader (sizingMethod='scale') if you want to support that browser.
Photoshop/Illustrator's 'Save for Web...' does not fully optimize. Perhaps my versions of Photoshop and Illustrator are too old, but I suspect this is still the case with the newer versions. I pretty much used these blind, assuming they were optimizing correctly. And don't get me wrong, it does a decent job, but its just not the best. Instead, run your images through Yahoo!'s smush.it site.
If you really do not need PNG-24, use PNG-8. PNG-8 is really a better GIF. But it is limited in color palette and transparency with respect to PNG-24. That being said, often you don't need the difference, especially for things like icons. When you can, use PNG-8 because you'll get much smaller file sizes.

That being said, you might think you need PNG-24 when you really don't. I did. I had these icons made that had full transparency. I knew, however, they were going to be on a white background, so I really didn't need all the transparency. Yet when I tried to save it as PNG-8, it just looked bad. The colors were all off. So it made me think that I needed PNG-24, but in reality it was Photoshop's optimization stuff that was being poor. In their defense, I wasn't helping them out by setting the white background ahead of time, which leads me to:
If you want to save a PNG-24 image as PNG-8, put in the background first. Once I made a white background layer, Photoshop then did a great job of saving it as PNG-8. And in fact, I could reduced the file size even more by using even less than 256 colors. Of course, I still had to run it through smush.it.
CSS sprites may reduce your page load (and image size further). CSS sprites are a way to group your images into one big file and then split them into separate files via CSS. There is a useful Web site to help you make them at csssprites.com. I couldn't figure out how to use it with my resizing requirements, but in the general case it should be at least tried, especially for icons where the color palette for your icons are similar. You get a win in image size. But you get a bigger win in reducing HTTP requests.
Custom icons are not that expensive. We got $40 custom icons and $10 recolored icons from iconshock.com. We talked to other icon designer firms as well, and prices were similar. Full disclosure: we created more than 3 icons (7), so we got a bit of a bulk discount. I did have a bad experience with iconeden.com, however. So I'd stay away from them.

For more image optimization tips, check out Yahoo!'s presentation.

Update: additional comments can be found here.