Recent writing on search and AI
-
AI Crawler Settings by Host and CDN: Cloudflare, Fastly, Akamai, WP Engine, Shopify, and More
Your host or CDN may block AI crawlers before they reach your site. Learn about defaults and available configurations for common ones.
-
Complete Reference On Allowing & Blocking AI Crawlers
Learn how LLM bots crawl your site and how to control the crawl. This comprehensive guide includes user agent lists for each LLM and what they do, a tool to check what your site is blocking and allowing, and the implications for search and LLM visibility.
I help organizations understand how people search, and how to be found.
I write and speak about the search engine industry and searcher behavior and help companies with online strategy and audience engagement. I’m fascinated by our searching culture and how it’s shifted the way we seek out and consume information. (And now it’s shifting again as LLMs reach the mainstream!)
Before this, I worked at Google, where I built Webmaster Central and helped launch sitemaps.org. I wrote Marketing in the Age of Google, founded Nine By Blue, and until 2026 was CEO of Keylime Toolbox.