I was reading Redfin’s Developer Blog and the IE blog a few months ago and I got this desire to write my own OpenSearch provider. OpenSearch was originally created by A9.com (an Amazon.com company) and was primarily designed as a way for web developers to publish search results in a standard and accessible format. This turns out to be a good idea because different types of content require different types of search engines. The best search engine for a particular type of content is frequently the search engine written by the people that know the content the best. Google is great at searching unstructured content on the internet, but when it comes to structured search on a single web site there are much better options (Endeca, FAST, Autonomy, Solr, my favorite SQL database, etc). The other benefit of OpenSearch providers is that it shifts the balance of power away from Google and back toward web browser vendors & web site developers.
Both of the major web browsers support the OpenSearch Referrer extension. IE 7+, Firefox 2+ & Chrome allows you to add search engines to your browser without leaving the web page. The best place to get started is from the browsers vendors themselves. You can add search providers from Microsoft’s site or you can add search providers from Firefox’s add-ons site. In the interest of full disclosure, Opera allows you to add search engines manually, and Safari currently does not support this feature in any form (unless you count using vi to edit the Safari executable or changing your OS’s hosts file as support, which I do not recommend).
Anyway, our developer friends at Redfin wrote a blog post about their OpenSearch provider on their dev blog some time ago. Of course, they took the easy way out by not developing an OpenSearch Suggestions extension (slackers). I decided that a search provider without suggestion support is lame, so I took a stab at creating one. I think what inspired me to write an OpenSearch suggestions provider is that the IE 8 team blogged about their new Visual Search feature (which embraces & extends the OpenSearch suggestions work that Firefox pioneered) and I could leverage the work to improve the search experience for both IE 8 & Firefox 2+ users. (And the satisfaction of having a cool feature that Redfin & Estately haven’t implemented yet was probably another factor).
This functionality is typically exposed to users, via the search engine bar, next to the address bar in your web browser. So in your page markup, you’ll add something like this that tells the browser that your web site has a search service.
<link title="RPA Real Estate Search" type="application/opensearchdescription+xml" rel="search" href="http://www.seattlehouses.com/Feeds/OpenSearch.ashx"/>
The above element points to your site OpenSearch Description XML file which describes your search service in a way the browser can understand. When you visit RPA’s site, the browser will read RPA’s OpenSearch Description file located here and unobtrusively let you add the site’s search providers.
Assuming everything is working correctly, the user should be able to visit RPA’s web site, click on the browser’s search bar to add our search provider like so… (IE’s screen captures are on the left, Firefox’s are on the right).
 |
|
I’ve also added a button in RPA’s search bar (see above right) in case site visitors don’t discover our search provider via the browser (I suspect most users would miss it otherwise).
After you’ve registered RPA’s search provider with your web browser, you can select it and just start typing. Since I’ve implemented a suggestions service, it will auto complete cities, school districts & neighborhoods as you type them (Didn’t I say this was cool?). I should note that although IE 7 & Chrome support OpenSearch, only IE 8 and Firefox currently support the suggestions providers. Anyway, if you wanted to look for listings in Bellevue, here’s what it currently looks like.
 |
 |
As you’ll notice, IE 8 & Firefox 3 displays suggestions differently on RPA’s site. This is intentional because IE 8 supports a newer version of the OpenSearch standards (Microsoft calls it Visual Search) and I designed RPA’s search provider to exploit this fact. In Firefox, the browser can only handle plain text suggestions, which can lead to ambiguous searches. For example, let’s say you search for Riverview. Riverview is both a neighborhood in Kent and a school district in Carnation / Duvall, so in Firefox there is no means for the user to tell the web site in which context they meant to search for when they typed in Riverview. I suppose one could create a “Did you mean” results page for cases like this, but I think that somewhat defeats the purpose of having suggestions support.
However, in IE 8, if a term has multiple contexts, the search provider can display them all and the user can select the one they meant. Also in IE 8, the search provider can display thumbnails next to the suggestions, which further helps the user quickly find what they are looking for. Although, I haven’t implemented that feature yet (mostly because I wasn’t sure what picture I should put up there for search terms that return multiple results), other web sites have. For example, if you wanted to buy a movie from Amazon or learn more about our 16th president from Wikipedia, the IE 8 search provider experience looks like this…
  |
As the Redfin developers stated, implementing OpenSearch Referrer extensions are surprisingly easy (so I think users will soon request them from all web sites once the word gets out). The OpenSearch Suggestions extensions are more difficult to implement because every single keystroke is essentially a REST web service call. If you aren’t careful, you could bring your web server to its knees real quick. However, given all the AJAX map based tricks today’s real estate web sites perform, this isn’t anything that a professional software engineer can’t handle.
Call me crazy, but I think OpenSearch providers are going to become bigger than RSS feeds over the next year. If IE 8’s forth coming release doesn’t launch them into the mainstream, I think future releases of Firefox & Chrome will improve upon IE 8’s good ideas. Maybe you should think of it as browser favorites on steroids? If search is sticky, then OpenSearch is superglue and duct tape. If Firefox’s suggestions support were the tip of the iceberg, then IE 8’s implementation is cooler than Barrow, Alaska. The future of OpenSearch looks bright, even if it’s cold outside.
That’s really interesting stuff Robbie… although I’m not sure I see how these targeted searches could be bigger than RSS in the near future.
I’ve had many of these site-specific search tools like IMDB, Amazon and Wikipediaon my firefox browser for years, and yet, I almost never touch them. I guess you could call it laziness, but if I’m looking for something specific to one of these sites (let’s say wikipedia), it’s just as easy for me to land on wikipedia and then do a google site search (which is pretty trivial thanks to my google toolbar). More times than not this answers my question pretty darn fast and reliably… and I’m not sure than any of these OpenSearch tools are doing something that is substantially better.
That’s really interesting stuff Robbie… although I’m not sure I see how these targeted searches could be bigger than RSS in the near future.
I’ve had many of these site-specific search tools like IMDB, Amazon and Wikipediaon my firefox browser for years, and yet, I almost never touch them. I guess you could call it laziness, but if I’m looking for something specific to one of these sites (let’s say wikipedia), it’s just as easy for me to land on wikipedia and then do a google site search (which is pretty trivial thanks to my google toolbar). More times than not this answers my question pretty darn fast and reliably… and I’m not sure than any of these OpenSearch tools are doing something that is substantially better.
It all depends on what you are searching for. When I want to buy a new book or movie, I go straight to Amazon in my search bar and bypass Google since I know Amazon will have better product information, let me buy it, sell it to me at a very competitive price, and ship it to me quickly for free (I love Amazon Prime). Google just wants to show me ads for Amazon’s lessor competitors. Sorry, Google is slowing me down in this case.
Similarly, if I wanted to look for houses on the market in Fremont, I’d go straight to RPA’s search provider (or perhaps Redfin’s), type in “Fremont”, since an NWMLS member will have more accurate data. On Google I need to type “Fremont, WA houses” before it presents me with page after page of out of date listings. Sorry, Google slows me down again in this case too.
I agree, if you don’t know where the best answer is, then Google is probably still is your best option. Also, keep in mind Google pioneered support for OpenSearch 1.1 suggestions and I suspect will implement OpenSearch 2.0 suggestions shortly after IE 8 gets released. Google is many things, but slow to copy good ideas isn’t one of them.
Put another way, if people want to improve the stickiness of their web sites and promote their brands (instead of Google’s), providing an easier way of getting at your site’s content is one way of doing it.
I have to agree with Robbie on this one. Though I still Google things from time to time like “Seattle Real Estate Blog” in case there is a new one I am missing, I usually go straight to the few blogs I know exist and read.
Once a homebuyer knows the area they want to buy in, they likely would follow the same logic pattern.
The term “Open Source” in and of itself, sounds like a movement to be supported. It’s a buzzword like “Transparency” or “Authenticity”. I’ll have to read the post a few times to understand that feature better. Robbie is most often way over my head, even when he is just chatting over dinner about technology 🙂
I have to agree with Robbie on this one. Though I still Google things from time to time like “Seattle Real Estate Blog” in case there is a new one I am missing, I usually go straight to the few blogs I know exist and read.
Once a homebuyer knows the area they want to buy in, they likely would follow the same logic pattern.
The term “Open Source” in and of itself, sounds like a movement to be supported. It’s a buzzword like “Transparency” or “Authenticity”. I’ll have to read the post a few times to understand that feature better. Robbie is most often way over my head, even when he is just chatting over dinner about technology 🙂
Robbie: I think we’re more in agreement than not.
I agree that if I’m looking for books, I’ll normally go straight to amazon (the one difference being that if I can’t find what I want using Amazon’s search, then I’ll normally do a google site search of Amazon since that often turns up better results). However, it sounds like neither of us think to go to the “opensearch” tool in the top-right of our browsers to search Amazon despite the fact that I would guess I’ve had that option in my browser for two years now.
Robbie: I think we’re more in agreement than not.
I agree that if I’m looking for books, I’ll normally go straight to amazon (the one difference being that if I can’t find what I want using Amazon’s search, then I’ll normally do a google site search of Amazon since that often turns up better results). However, it sounds like neither of us think to go to the “opensearch” tool in the top-right of our browsers to search Amazon despite the fact that I would guess I’ve had that option in my browser for two years now.
Ardell gets it – even if she doesn’t think she does :). The only reason people use Google, is because your brand & online experience isn’t strong enough for people to visit you in the first place instead.
It could be that it’ll take 2 or 3 years for Visual Search / OpenSearch 2.0 to become as common as RSS is today. It might be like Ajax was, and doesn’t take off until somebody other than IE 8 (which admittedly hasn’t been released yet) comes up with a solid implementation. I do think it will happen eventually though.
Ardell:
I don’t think either of us are talking about using google to find websites, but rather searching for information on a specific site. In other words, here are three ways you *could* search amazon for a particular book or a particular topic:
1) Go to Amazon.com and use their search tool. (this is the most common method).
2) Use a opensearch tool that *might* live in the top right of your browser (this really depends on the browser your using and whether or not you’ve installed the Amazon specific tool).
3) Do a google site search. Because I have the google toolbar installed, this is as simple clicking the “search site” button while I’m on a particular site (such as amazon).
Methods 1 and 2 will give you identical results since they both use Amazon’s home-grown search engine while your results will often be different if you choose to search the Amazon site using Google.
To give you an example, here are the results for moving to Seattle on Amazon using Amazon’s search engine (method 1 or 2) and here are the results for moving to Seattle on Amazon using google’s site search (method 3).
I’m not even recommending the third method over the first two because Amazon’s search is pretty darn good. But most sites don’t have the search algorithm backends to be any good and I’d take Google’s site search over their home-grown search almost every time.
One major exception is real estate search, where refining searches based on specific fields can be extremely useful. In that case, like Robbie, I’d go directly to a real estate search site, as oppose to use an opensearch tool, because those sites almost always give me the ability to filter things in a much more intuitive way than a single input field can.
Ardell:
I don’t think either of us are talking about using google to find websites, but rather searching for information on a specific site. In other words, here are three ways you *could* search amazon for a particular book or a particular topic:
1) Go to Amazon.com and use their search tool. (this is the most common method).
2) Use a opensearch tool that *might* live in the top right of your browser (this really depends on the browser your using and whether or not you’ve installed the Amazon specific tool).
3) Do a google site search. Because I have the google toolbar installed, this is as simple clicking the “search site” button while I’m on a particular site (such as amazon).
Methods 1 and 2 will give you identical results since they both use Amazon’s home-grown search engine while your results will often be different if you choose to search the Amazon site using Google.
To give you an example, here are the results for moving to Seattle on Amazon using Amazon’s search engine (method 1 or 2) and here are the results for moving to Seattle on Amazon using google’s site search (method 3).
I’m not even recommending the third method over the first two because Amazon’s search is pretty darn good. But most sites don’t have the search algorithm backends to be any good and I’d take Google’s site search over their home-grown search almost every time.
One major exception is real estate search, where refining searches based on specific fields can be extremely useful. In that case, like Robbie, I’d go directly to a real estate search site, as oppose to use an opensearch tool, because those sites almost always give me the ability to filter things in a much more intuitive way than a single input field can.
Robbie:
You raise an interesting question… If you were doing a search for real estate in the Seattle area, do you think you’d have a better experience if you went directly to RPA’s site or if you used the OpenSearch widget that you created for their site?
(By the way, I tried it out and it worked flawlessly… but still, when I use it, after my first search, I’m now on the RPA site using their search tools, and (assuming I was a normal consumer), I would just return to their website each time I wanted to do an refined home search.)
Robbie:
You raise an interesting question… If you were doing a search for real estate in the Seattle area, do you think you’d have a better experience if you went directly to RPA’s site or if you used the OpenSearch widget that you created for their site?
(By the way, I tried it out and it worked flawlessly… but still, when I use it, after my first search, I’m now on the RPA site using their search tools, and (assuming I was a normal consumer), I would just return to their website each time I wanted to do an refined home search.)
I’m with Dustin on this one. While the idea is fantastic for A LOT of industries, real estate is an exception. The goal for people looking for real estate websites is to search for property. Once they find a site which has a great real estate search, the opensearch is left out in the cold. The opensource is an extra step in which the customer will skip once the target website is found.
To not seem to pessimistic, the idea could be a homerun in other industries. Areas where customers are not searching for a needle in a haystack would be a great target for this.
Dustin,
I don’t think it’s a simple either/or proposition. If you aren’t already on the site, the opensearch provider is far more convenient because you don’t have to navigate to the site first, in order to be able to search on it. Having a search bar in the browser (regardless of the site its pointing to), is just so dang convenient.
Once you’re on the site, I suspect some users would use the search refinement tools on the site (because it’s more powerful), and others would continue to use the browser search bar instead (because it’s easier to use). I just want the visitors to enjoy their time on site enough to consider doing business with my client and/or visit the site again.
I think you bring up an excellent point in that most sites don’t even have text search backends (let alone good ones). This might be why Google wins by default so often.
I also think part of our differences are due to that you’ve only seen the present implementation of OpenSearch 1.1 in Firefox, and I suspect you haven’t yet played around with the forthcoming implementation of OpenSearch 2.0 in IE 8. The newer standard leads to a more compelling in browser search experience and I find I’m using it more & more.
I think the other problem is that in the big scheme of things, very few sites even support the OpenSearch 1.1 suggestions standard (let alone the draft 2.0 standards). Most sites that do support it, only support OpenSearch 1.0 referrer standards (which as you point out, isn’t that compelling).
For example, Amazon doesn’t have suggestion support in Firefox, so I can understand why you think the experience isn’t compelling vs Google (because in Firefox, Google does have suggestion support & Amazon doesn’t). It’s entirely likely that Google will be among the first sites to exploit this newer technology.
Hmm….. very interesting. I like the concept, never heard of these other search engines. With Google containing the wide share of the SE market I would like to see them implement this…
Todd,
Not everybody uses only 1 web site to find something. When I search, I tend to use multiple “good ones” and not just one site. It might be I just haven’t found “the one good one” yet, I like second opinions, or perhaps in the greater Seattle area we are blessed with many good ones.
I do think OpenSearch support is a feature that all good ones will soon have. Besides, Redfin already has OpenSearch 1.0 support (and I suspect after they read this blog post, they’ll consider adding 1.1 or 2.0 support to their backlog 🙂
Clint,
You haven’t heard of Amazon & Wikipedia, or you haven’t heard of OpenSearch? OpenSearch is a technology standard (like HTML or RSS) and not a search engine per se. Search engine companies are very likely to take advantage of the technology. In fact, Google, Yahoo & Microsoft all support OpenSearch 1.1 (suggestions). Although, of “the big 3” currently only Microsoft supports the newer OpenSearch 2.0 standard (visual suggestions & urls with suggestions). However, I suspect Google will embrace OpenSearch 2.0 eventually (since they were one of the first to embrace 1.1 suppport).
Todd,
Not everybody uses only 1 web site to find something. When I search, I tend to use multiple “good ones” and not just one site. It might be I just haven’t found “the one good one” yet, I like second opinions, or perhaps in the greater Seattle area we are blessed with many good ones.
I do think OpenSearch support is a feature that all good ones will soon have. Besides, Redfin already has OpenSearch 1.0 support (and I suspect after they read this blog post, they’ll consider adding 1.1 or 2.0 support to their backlog 🙂
Clint,
You haven’t heard of Amazon & Wikipedia, or you haven’t heard of OpenSearch? OpenSearch is a technology standard (like HTML or RSS) and not a search engine per se. Search engine companies are very likely to take advantage of the technology. In fact, Google, Yahoo & Microsoft all support OpenSearch 1.1 (suggestions). Although, of “the big 3” currently only Microsoft supports the newer OpenSearch 2.0 standard (visual suggestions & urls with suggestions). However, I suspect Google will embrace OpenSearch 2.0 eventually (since they were one of the first to embrace 1.1 suppport).
BTW – IE 8 was released this week. If you are still using IE 6, what’s wrong with you?! 🙂
Robbie,
We received a notice from the mls that we can’t integrate with 8 yet…but they are working on it.
Pingback: Opensearch vs custom toolbar vs smart keyword vs bookmarklet (II) | Musing about librarianship
I agree, I think Open Search definitely has the capability to evolve into something larger than RSS but I suspect it’s going to be used more in the mashup world.
RSS has always been kind of a geek centric technology since it’s presence is kind of invisible in your typical web browser. I think Visual Search, Web Slices, & Accelerators other newer IE 8 technologies are probably more end-user accessible and more obvious.
I think it’s ultimate success will depend if IE’s competitors copy IE 8’s good ideas and how many web developers support it. Maybe it’s doomed to be a Microsoft only web technology for 5 years (like AJAX was), until somebody else somebody names it after a household cleanser and supports it on another browser?