Sphinx search engine news

Public repo, Drizzle support, and call for Sourceforge voters

shodan — Wed, 27 May 2009 02:49:27 +0200

I was initially going to call this quick entry "All the small things" but then talked myself into keeping it informative anyway. Then again they seem small to me, but might be huge for someone else, eh?

First, we now have a public Subversion repository. It's a read-only mirror hosted at Google Code, the project name is sphinxsearch. Changes are currently svnsync'ed hourly. Likely to add a post-commit hook a bit later. Disclaimer: I did not know we could do it that easily, only learned about the possibility of syncing earlier today. (Thanks fly out to Alexey Vinogradov.)

Second, Eric Day made a patch that enables Sphinx support for Drizzle - thanks a lot! Chances are we're integrating this into 0.9.9-release, when time permits; the patch looks fine at a glance, but I still want to review and actually try it for myself before committing.

And third, last but not least, I just realized that Sourceforge Community Awards 2009 are not only in progress, but actually closing nominations phase on May 29th (if not by May 29th) ie. there's less than two days left. So if you feel that Sphinx is worth the Community pick, you can vote here right now. SysAdmins, Developers, and Enterprise all seem fitting categories to me. Curious if we can at least make it through nominations with almost no time left; according to CCA schedule, we'll all know that in a month. Thanks in advance to all voters!

Sphinx 0.9.9-rc2 is out

shodan — Wed, 08 Apr 2009 00:04:49 +0200

Sphinx 0.9.9 branch is officially feature-frozen and we are making 0.9.9-rc2 available. As those who've been following 0.9.x releases should be accustomed to, despite a modest rc1 to rc2 tag change it adds about 30 new features once again. (Wonder if we'll be able to break the spell in 1.x and be less greedy about the version increments.)

The ultimate new feature couple is MySQL binary protocol and SphinxQL query language. Meaning that searchd can now pretend it's mysqld. Meaning that you can use ye good olde mysql command-line client to connect to searchd and fire your queries using regular SELECT syntax! More details are available via that link above.

But wait, there's more. We added ODBC support both on Windows and Linux (through UnixODBC) so you now can connect directly to that Oracle or MS SQL database. We added a bunch of performance counters, IMO they look especially nice when viewed using SHOW STATUS via mysql client program. Strict order operator (aaa << bbb) and field-start and field-end keyword modifiers (^hello world$) were added to full-text query language. GROUP BY now supports aggregate functions (AVG, MAX, MIN, and SUM), accessible both through newly added SphinxQL interface and SetSelect() API call.

Other 20+ features are smaller and these 7 most major ones should be enough to encourage you to get and try it anyway. For the curious, 0.9.9-rc2 change log lists them all along with the bug fixes.

How safe is it to upgrade? Generally, both API and indexes are backwards compatible (for about 2 years now); early adopters roll out intermediate non-public releases on production systems on a rather regular basis; and we never publish releases with known major general issues. And 0.9.9-rc2 is no exception. Of course nobody (except Donald Knuth) can guarantee that his software is absolutely bug free. But what we can guarantee to our customers is that we'll priority fix all bugs that you can repeat. (The keyword here is "priority" as we're gradually fixing all the bugs reports we receive.) So if you're looking for safer production deployment of all the shiny new features, consider support packages. They perfectly cover 0.9.9-rc2.

What's next? We will be fixing issues discovered in rc2 for about 1-2 months from now on before announcing stable release - depending on your feedback. Do send us some! In the meantime, the work on real time updates is progressing, we'll be showing those at MySQL UC 2009. That said, next two planned releases are 0.9.9-release and 1.10.1-alpha, with the one which is the le... the bigger of two goods coming out first.

Meet the Sphinx (in LA or SF)

shodan — Tue, 24 Mar 2009 22:58:07 +0200

MySQL Conference 2009 is due in less than a month now, I am attending again, and this means a number of Sphinx-related events in the US:

Joe Devon arranged a Los Angeles MySQL UG meetup on April, 17th and I will be giving a short talk on Sphinx there.
We'll be showing Sphinx at DotOrg booth at MySQL Conference 2009 on April 21st and 22nd once again, drop by booth TT12.
I will be giving two talks on Sphinx at Percona Performance Conference 2009 event co-hosted by Percona, MySQL, and O'Reilly. Those are preliminary scheduled on April 22nd and 23rd respectively.

I will be in LA area from April 16th to 19th, then in SF Bay area from 20th to 28th, then off to Vegas for 3 days. Might have a free day or two both at LA and SF, and don't mind if you show us a secret nice lunch place in Vegas either, so let me know if you'd like to meet. (Slim tall blondes will be given higher priority.)

More Sphinx services

shodan — Sat, 14 Mar 2009 23:31:57 +0200

Starting today we're offering a new and shiny Sphinx service. Namely, support packages. We've been funding all the development from our consulting services for about 2 years now. However as the consulting is charged by the hour, using it on the support and fixes seemingly resulted in a conflict of interest, as from the customers' point of view it kind of gave us the incentive to introduce more issues and spend more time fixing them. (In fact it never did of course, but that's another story.) Anyway, support packages introduce unlimited email support and unlimited repeatable bug fixes for a flat fee, so even the seeming conflict is no longer. We are also bundling a number of free consulting hours that you can use on anything you like - to name a few possibilities search implementation, performance review, new Sphinx public feature development, or even custom development for your application. Last but not least top support level includes a 24x7 emergency hotline with direct access to experts.

But what has been happening on the tech side during the last months? Actually a whole lot. 0.9.9-rc2 will introduce over 20 new features again, including several really major ones. It is nearing a code freeze and being documented right now. A public release is due shortly. Also work on realtime updates (for version 1.x) is now underway, sponsored by BoardReader for March and April. The effort is pretty huge and we're going to need additional funding for RT Updates starting May, so if your company's interested in making them happen at all and getting priority support for the feature at the earliest stage, as well as other perks, drop me a note.

Sphinx goes billions

shodan — Wed, 14 Jan 2009 04:39:51 +0200

To my constant amusement, it's not only Sphinx code but also the website that needs some work every now and then, so here's the update.

I've added FAQ section. It's not big yet (only 8 questions covered), and the answers are typically terse, but a start is a start.

Presentations section has a couple new talks. One small step for the page but giant leap for me is that all the previously posted talks were given by ourselves. And now we're not alone - hooray and cheers for Ralf and Mike!

Last but not least, Powered By section, now at 113 sites and counting, was updated and restyled. I had long wondered how much Sphinx search queries are performed per month if we sum all the sites using it, and whether we already hit 1B page views per month or not. Being open-source, there's no easy way to tell. But now with the addition of craigslist to Powered By list I finally know that we do. Many thanks to Jeremy Zawodny who worked hard on making that happen, my itch is no more. :-)

Billions of documents indexed, billions of queries per month. Mind-blowing...

0.9.9-rc1 is out

shodan — Mon, 17 Nov 2008 22:08:08 +0200

All the new features just have been finally documented, all the internal tests pass, so here goes another major step towards version 1.0 (or maybe 1.10 right away? or even 3.23?). Meet a beta version of Sphinx 0.9.9, introducing 34 new features; a handful of improvements and fixes worth a mention in the changelog; and numerous minor changes I did not count. Choosing top-10 features to highlight here was actually a tough job. (Even though choosing top-3 was a no-brainer.) Here they go.

added select-list feature w/full expressions support
added arbitrary brackets/negations nesting support to query language
added config reload on SIGHUP
added signed 64bit attrs support (sql_attr_bigint directive)
added persistent connections, UNIX-socket, and multi-interface support (Open(), Close(), listen)
added kill-lists support
added MS SQL (aka SQL Server) source type support
added inplace inversion of .spa and .spp (inplace_enable, 1.5-2x less disk space for indexing)
added multiforms support (multiple source words in wordforms file)
...and 20+ other features in the changelog

Normally this major changeset would be declared an alpha. However, thanks to brave testers, 0.9.9 already runs in production on several sites. And also its test coverage is actually even better than that of 0.9.8. So we can light-heartedly call it beta (as in "no known major bugs").

Bundled website update includes a few extra links, of which I personally would claim C# .NET API port and Ubuntu package (in Contributed software section) most interesting. And, as you see, we've also refit News and Docs sections; you now can (anonymously) comment on the posts and submit 'em to places like Digg. Testing time!

0.9.8.1-release is out

shodan — Thu, 30 Oct 2008 07:18:00 +0200

Today's Downloads star is Sphinx 0.9.8.1. This is a bugfix release, and every commit received especially careful review, so upgrades from 0.9.8 should be smooth. Most of the issues were actually pretty subtle and only occurred under (combinations of) special, and rather rare, circumstances. Except for the index merge feature, where several major bugs were reported in about a month since 0.9.8 release (and then it took another month to get them all caught, reproduced, and fixed). The complete change log is available in the documentation.

Speaking of months, we've been silent (on the website) for a while. Not really intentionally, rather just because a whole lot of things happened in the meantime, making me uber-busy, and thus postponing the release rollout somewhat past the point when it was due. For one, October was especially busy; in late September the update rollout was scheduled for October, 1st, but.. you see the calendar. I'd rant about all the events leading to this much of a deadline slip if they were not that numerous. On the brighter side, the update includes more fixes this way.

And speaking of events, here goes a new section called Presentations. For now, it only carries our own talks; but please submit yours too! I've just added new slides from 2 conferences held this fall in Russia that I talked at. Going to translate the slides to English as time permits. Some slides back from 2007 are also there (marked as new based on the time of addition, not the talk).

Community contributors also did a bunch of work in the meantime, resulting in a couple of new localized articles (in Italian and Turkish), and more contributed software - namely, a patch that adds Firebird support, Drupal modules, Haskell API port, and PECL API port. Of these, I'd especially like to highlight PECL API port. It's based on libsphinxclient (bundled with Sphinx), works as a drop-in replacement to the reference PHP API, and is naturally faster (though we've yet to see the benchmarks). Cheers to Tony, Pierre, Johannes, Markus, Chris, Eelco, Ludovico, and Salih!

Last but not least, likely the final teaser for 0.9.9. It adds a number of shiny new features, it now runs in production on several sites, and the only thing that currently holds the alpha release from going public is actually simply the lack of documentation on those features. Ah, and it is where the most changes went, which is one of the reasons why the change log for 0.9.8.1 is so small.

0.9.8-release is out

shodan — Tue, 15 Jul 2008 19:25:00 +0200

I have been frequently put to shame for Sphinx release schedule and version numbering. Indeed, taking into the account the typical amount of features and fixes per snapshot, most if not all of so-called snapshots in 0.9.8 branch could likely qualify for a separate release, receive their very own minor version tags, etc.

We are going to work on that. But even the longest journey begins with a first step. Time to make that step.

Over 15 months, over 700 revisions, and about 70 new features since the previous "release" tag, Sphinx 0.9.8-release is finally out. As usual, source tarball and Win32 binaries are available in the Downloads section.

The complete list of new features since 0.9.7 is available in the documentation. With 70 entries, it's too big to copy here, and had to be split in several sections for the doc. It does not include bugfixes at all, cause that would make it too big even for the doc. (Also, I'm too lazy to go through all the commits and summarize all the fixes we did over the year).

The complete changelog since RC2 is also in the documentation. (This one includes the fixes.) There were no new features in the engine itself since 0.9.8-rc2, because we intentionally froze it. Nevertheless, we did some additions to the whole package:

added Ruby API and pure C API;
added network timeout and error parameters to PHP API;
added sphinx-min.conf sample file;
added --without-iconv switch to configure.

Some trivia about 0.9.8-release:

0.9.8 is revision 1371;
0.9.7 is revision 606;
0.9.8 source size is almost precisely 2 times the 0.9.7 source size;
automated testing suite runs 33 tests, 250 subtests (meaning 250 different config variants), and verifies 1695 queries against manually checked reference results;
creating the unified documentation, ie. documenting all the sphinx.conf features and API calls, took 5 all-night-long bursts to complete (and then some daylight polishing that does not count);
the only bugfix mentioned in the list of changes since 0.9.7 is the fix that enabled Sphinx to run on SPARC and ARM.

Also in the news, we're among the finalists in SourceForge Community Choice Awards 2008! Sphinx is in the shortlists for “Best Project”, “Best Project for the Enterprise”, and, somewhat unexpectedly for me, “Most Likely to Be the Next $1B Acquisition”. Many thanks to all who voted for Sphinx during the nominations! Now is the time to vote again in the 2nd round. There are several quite popular competitors (such as OpenOffice), so please spread the word as much as you can.

As promised, the next update is going to be 0.9.9 alpha. We've already got some nice new features there for you, but need to merge all the fixes from 0.9.8 and eliminate some known crashes. In the meantime, have fun with 0.9.8, be sure to report everything that resembles a bug, and vote for Sphinx again. Thanks!

PHPconf '2008 talk slides; SF CCA '2008 call-for-voters

shodan — Fri, 06 Jun 2008 21:35:00 +0200

The final release of 0.9.8 and an alpha of 0.9.9 are still in progress, but there's a number of small things that yearn to be published already. Hence this website only update.

First, here go the Sphinx talk slides from PHPconf 2008, both the original Russian version (PPT, 460 KB), and English version (PPT, 458 KB) (alas, the translation will probably miss some jokes). The event was recorded, and it was told that the talk videos (also in Russian) will be available at some point, but at the moment they are not.

Second, Arnold D. just pointed me to SourceForge Community Choice Awards 2008. You can nominate Sphinx in several different categories; I think that “Best Project”, “Best Project for the Enterprise”, and “Best New Project” do possibly apply. (Not that I'd violently reject $1B acquisition offer, of course.. but it also mentions 'likely' so let's try to stay realistic here.) The nominations will close in mid-June, so please hurry.

Finally, there are several new articles on Sphinx, and also a few more web sites who decided to acknowledge that they're Sphinx-powered. “Powered by” list is already getting hard to scroll but is still slightly under 100-site lets-redesign-it internal threshold; so you're especially encouraged to submit yours.

MySQL UC talk; PHPconf talk; and dev plans

shodan — Fri, 16 May 2008 18:51:00 +0200

The slides from the Sphinx talk that I gave a month ago at MySQL UC 2008 had already been available from O'Reilly for some time. However, I've just wrapped up the slides with the talk itself. The resulting article combines both basic overview and some of the advanced stuff on the searching internals; so I hope it should be an interesting read for everybody. Here's the article in Word format (131 KB) and PDF format (202 KB), and for completeness, here are the slides in PPT format (638 KB) (they are embedded in the article).

I'll also be giving a talk on Sphinx at PHPconf 2008 Russia in Moscow (on May 30th), so if you're speaking Russian and interested in Sphinx, catch me there.

Most of the April was spent conferencing so there's been less work on Sphinx than usual (and no updates on the site at all). However we had written some code in the meantime anyway, mostly in 0.9.9 branch. There were a few fixes in 0.9.8 since RC2 but not really much; so I am planning to finish the fixing and roll out 0.9.8-release out within a couple of weeks. Alpha version of 0.9.9 will be made available in a few days after that.