Comparing in-site search engines

A long time ago, I took a fairly rigorous look at in-site search engines (the kind that you can integrate into a website to search just that site). I compared a bunch of tools, and came up with a fairly clear front runner: Spiderline. (For a blast from the past, check out my search engine article from 2004)

Spiderline costs a bit of money (it starts at about $8/month), but it offers seamless integration into your site, and a lot of control. I've implemented for a bunch of clients with nary an issue. I never found Google's old site search compelling - it was just impossible to make it look like a Google search glommed onto your website, and look and feel was typically important to clients.

But Google has a new offering now: Google Co-op custom search, which allows some new options. I had the opportunity to compare Google to Spiderline both for a consulting client, and for the Idealware site itself. A comparison chart is below. They're both good options, with some interesting tradeoffs.

Note that I haven't done a broad survey of the search tools market in years, and there may be other great, or better, search tools that I don't know about.

	Google Custom Search Engine	Spiderline
	http://google.com/coop/cse/	http://www.spiderline.com
Price	Free.	Based on number of site pages and number of crawls per month. For instance, 5 crawls a month for a site with less than 500 pages is $10/month.
How often does it index your site?	About monthly, maybe more for high volume sites. Not controllable.	Up to daily; more frequent indexes cost more.
What pages are indexed?	The pages that Google indexes in the main search engine, for whatever reason Google decides to do that. Real risk that it might not include key pages, especially for low volume sites. Have to do hacks to your site to allow Google to index pages under password protection	Whatever you want, including most pages under password protection, depending on your protection method
Look and feel	Mostly customizable to match to your site. Uses iframes, which concerns me that implementations might have oddities	Complete control over every aspect of look
Search result quality	Google quality. Notably better than Spiderline for sites I looked at.	Reasonable, but less good at putting the best links on top. And seemed to include a lot of results that weren't relevant.
Info for search results listing	Nice display of page results with very little required of website metadata (it works some Google magic to pull the page title out of the body copy if there's no metadata). Less control over what is shown.	Needs strong page title and metadata for every page to be optimal. But if you have good metadata, you have complete control over the results, and can display the metadata title and description on the results
Customer support	Community forums only	Phone or email, real, actually helpful humans
Effort to format, integrate into site	Half day to a day	Half day to a day

It comes down to, in my mind, a tradeoff between price and search results quality vs. control over what's indexed and how often it's indexed

So what was the answer? My client decided on Google. They get a fair amount of traffic, are well represented in the main Google search engine, and don't have a lot of critical updates, so it was a great choice for them. And as they have a lot of pages on their site, the price for Spiderline was non-negligible.

We choose Spiderline for the Idealware site. I miss the Google quality of search results, but for us it was a deal killer to have to wait a week or more for Google to index our new articles. If you search for a phrase that you can see right there on the homepage (in a new article blurb), and get no results, the search just seems broken. And as the Idealware site isn't very big, the monthly price for Spiderline is only $10/month - no big deal for a good search on the site.