There you go again – the MLS doesn’t scale

Posted on by
16

[photopress:Reagan.jpg,thumb,alignright]Ever since Zearch, I’ve been bombarded with work to update or create MLS search web sites for various brokers & agents across the country. Because of this, I’ve had the opportunity to deal with another MLS in the Bay Area (EBRDI) and Central Virginia (CAARMLS). Before I begin another MLS rant (and cause the ghost of Gipper to quip one of his more famous lines), I want to say the IT staff at both EBRDI & the NWMLS have been helpful whenever I’ve had issues, and this primary purpose of the post is to shine a light on the IT challenges that an MLS has (and the hoops that application engineers have to jump through to address them).

After working with EBRDI, and the NWMLS, I can safely say the industry faces some interesting technical challenges ahead. Both MLSes have major bandwidth issues and the download times of data from their servers can be so slow, it makes me wonder if they using Atari 830 Acoustic modems instead of network cards.

The EBRDI provides data to members via ftp downloads. The provide a zip file of text files for the all listing data (which appears to be updated twice daily), and a separate file for all the images for that day’s listings (updated nightly). You can request a DVD-R of all the images to get started, but there is no online mechanism to get all older images. This system is frustrating because if you miss a day’s worth of image downloads, there’s no way to recover other than bothering the EBRDI’s IT staff. If the zip file gets corrupted or otherwise terminated during download, you get to download the multi-megabyte monstrosity again (killing any benefit that zipping the data might have had). Furthermore, zip file compression of images offers no major benefit. The 2-3% size savings is offset by the inconvenience of dealing with large files. The nightly data file averages about 5MB (big but manageable), but the nightly image file averages about 130 MB (a bit big for my liking considering the bandwidth constraints that the EBRDI is operating under).

As much as I complain about the NWMLS, I have to admit they probably have the toughest information distribution challenge. The NWMLS is probably the busiest MLS in the country (and probably one of the largest as well). According to Alexa.com, their servers get more traffic than redfin or John L Scott. If that wasn’t load enough, the NWMLS is the only MLS that I’m aware of that offers sold listing data [link removed]. If that wasn’t load enough, they offer access to live MLS data (via a SOAP based web service) instead of daily downloads that the EBRDI & CAARMLS offer their members. If that wasn’t enough load, I believe they allow up 16 or 20 photos per active listing (which seems to be more than the typical MLS supports). So, you have a database with over 30,000 active listings & 300,000 sold listings, all being consumed by over 1,000 offices and 15,000 agents (and their vendors or consultants). The NWMLS also uses F5 Network’s BigIP products, so they are obviously attempting to address the challenges of their overloaded information infrastructure. Unfortunately, by all appearances it doesn’t seem to be enough to handle the load that brokers & their application engineers are creating.

Interestingly, the other MLS I’ve had the opportunity to deal with (the CAARMLS in Central Virginia) doesn’t appear to have a bandwidth problem. It stores it’s data in a manner similar to EBRDI does. However, it’s a small MLS (only 2400-ish residental listings) and I suspect the reason it doesn’t have bandwidth problem is because of the fact it has fewer members to support and less data to distribute than the larger MLSes do. Either that, or the larger MLSes have seriously under invested in technology infrastructure.

So what can be done to help out the large MLSes with their bandwidth woes? Here’s some wild ideas…

Provide data via DB servers. The problem is that as an application developer, you only really want the differences between your copy of the data and the MLS data. Unforunately, providing a copy of the entire database every day is not the most efficient way of doing this. I think the NWMLS has the right idea with what is essentially SOAP front end for their listing database. Unfortunately, writing code to talk SOAP, do a data compare and download is a much bigger pain than writing a SQL stored proc to do the same thing or using a product like RedGate’s SQLCompare. Furthermore, SOAP is a lot more verbose than the proprietary protocols database servers use to talk to each other. Setting up security might be tricky, but modern DB servers allow you to have view, table, and column permissions so I suspect that’s not a major problem. Perhaps a bigger problem is that every app developer probably uses a different back-end, and getting heterogeneous SQL servers talking to each other is probably as big a headache as SOAP is. Maybe using REST instead of SOAP, would accomplish the same result?

Provide images as individually down-loadable files (preferably over HTTP). I think HTTP would scale better than FTP would for many reasons. HTTP is a less chatty protocol than FTP is, so there’s a lot less back & forth data exchange between the client & server. Also there’s a lot more tech industry investment in the ongoing Apache & IIS web server war than improving ftp servers (I don’t see that changing anytime soon).

Another advantage is that most modern web development frameworks have a means of easily making HTTP requests and generating dynamic images at run time. These features mean a web application could create a custom image page that downloads the image file on the fly at run-time from the MLS server and caches it on the file system when it’s first requested. Then all subsequent image requests would be fast since they are locally accessed and more importantly, the app would only download images for properties that were searched for. Since nearly all searches are restricted somehow (show all homes in Redmond under $800K, show all homes with at least 3 bedrooms, etc), and paged (show only 10, 20, etc. listings at a time), an app developer’s/broker’s servers wouldn’t download images from the MLS that nobody was looking at.

Data push instead of pull. Maybe instead of all the brokers constantly bombarding the MLS servers, maybe the MLS could upload data to broker servers at predefined intervals and in random order. This would prevent certain brokers from being bandwidth hogs, and perhaps it might encourage brokers to share MLS data with each other (easing the MLS bandwidth crunch) and leading to my next idea.

BitTorrents? To quote a popular BitTorrent FAQ – “BitTorrent is a protocol designed for transferring files. It is peer-to-peer in nature, as users connect to each
other directly to send and receive portions of the file. However, there is a central server (called a tracker) which coordinates the action of all such peers. The tracker only manages connections, it does not have any knowledge of the contents of the files being distributed, and therefore a large number of users can be supported with relatively limited tracker bandwidth. The key philosophy of BitTorrent is that users should upload (transmit outbound) at the same time they are downloading (receiving inbound.) In this manner, network bandwidth is utilized as efficiently as possible. BitTorrent is designed to work better as the number of people interested in a certain file increases, in contrast to other file transfer protocols.”

Obviously MLS download usage patterns match this pattern of downloading. The trick would be getting brokers to agree to it and doing it in a way that’s secure enough to prevent unauthorized people from getting at it. At any rate, the current way of distributing data doesn’t scale. As the public and industry’s appetite for web access to MLS data grows and as MLSs across the country merge and consolidate, this problem is only going to get worse. If you ran a large MLS, what would you try (other than writing big checks for more hardware)?

“2010 or Bust”

Posted on by
30

This may be the new mantra of the DOJ when it comes to its current anti-trust case against the National Association of REALTORS.  The boys at Freakonomics are at it again with an interesting post about some inside scoop on settlement discussions.

In a recent presentation [link removed] to some MLS folks, I concluded that the DOJ has absolutely no incentive to settle this case unless NAR comes to its knees on the issue of who is a “broker” for purposes of access to MLS listing data.  I also believe that this case is very analogous to the Visa/Mastercard case brought by the DOJ in 1998.  That case took 5 years to reach its conclusion.  Based on that same time line, the NAR case could easily stick around until 2010.

Every day that goes by without resolution of this case helps the new real estate players gain necessary traction in their quest to legitimacy.  If NAR settles the case and gives the DOJ what it wants (open access to all “brokers”), Katy bar the door.  If NAR holds out and we don’t have resolution until 2010, then (regardless of the end ruling in the case), the practical effect may likely be the same with these “new” models then being accepted as part of the real estate brokerage industry. 

This case, like the Visa/Mastercard case, is about innovation or maybe the lack thereof.  Query the impact if the human and monetary resources that are being used to defend this case were instead re-directed to innovate on behalf of traditional brokers to leverage the riches of data that lies within its control.   

-Russ   

Coldwell Banker Bain’s new interactive map search launched today

Posted on by
4

Very similar to what John L Scott did (they were both done by the same dev company) but a little more localized as far as neighborhood drill-down. They also added sold property listings driven by address reference rather than map based, but still pretty handy.

As I’m an agent for Coldwell Banker Bain I’ll refrain from singing my praises of the improvements. I’m a little more interested in what everyone else thinks of it. Check it out at http://www.cbba.com/InteractiveMapSearch.aspx

No more Redfins, no more Zillows or Trulia’s — no more innovation!

Posted on by
12

Ok, I’m a little off the real estate target here but it is an issue that effects us all. The biggest factor behind the creation of innovative companies like CraigsList, eBay, Amazon.com and even Google is what? The Internet of course. The Internet has been a decisive factor in leveling the playing field between the giant corporations and the startups with great ideas. This provides the environment for real estate related companies like Redfin and Zillow as well as social networking sites such as MySpace.com and YouTube.com.

Apparently many of the telcos (Sprint, Vorizon, AT&T, etc) feel that they should be getting a share of the revenue created by these new ideas. In exchange for creating a faster Internet, their proposal is a toll booths, private express ways and the slow lane for those who don’t want to line the pockets of the telcos. The very same pockets that have already been lined with $200 Billion in tax breaks to create an infrastructure they’ve failed to deliver on already.

So what to the power hitters and luminaries of the industry have to say? Well, on one side we have AT&T CEO Ed Whiteacre who said “that in order to bring America up to speed through fiber-to-the-premises (fttp) wiring, content providers are going to have to pony up to use his “pipes.” I’m far more inclined to believe the likes of Craig Newmark, founder of CraigsList.

‘Imagine if the leaders of 16th century Germany, feeling threatened by the democratizing forces of the printing press, had taken Gutenberg’s invention and limited its use to those they politically agreed with — or if Luther had to pay licensing fees for nailing up his 95 Theses on every church door in Germany,’ writes Craig Newmark in an opinion column in the San Francisco Chronicle.

Sir Tim Berners-Lee, father of the Internet, has characterized this as a US-only problem at present. “In Europe, net neutrality is the rule“ he said. Interestingly enough they are also way ahead of the US in broadband technology and capacity. What else can you say about an issue that has united Microsoft, Google, Yahoo and Moveon.org all on the same side.

Some of the proposed Net Neutrality legislation is seeking to prevent the telcos from charging extra tariffs based upon the type of content delivered. But even with the heavy hitters like Microsoft and Google, this initiative is facing a tough battle. Just last week those companies favoring net-neutrality lost a key vote in Washington DC. The key lobbyists seem to be siding with the telcos and their bread-and-butter lobbying dollars.

On the other hand, is this law really necessary? There are some who advocate letting them put up road blocks — it will spur more market forces to alternative technology like Wi-Max. Others are against the net neutrality laws as they are currently proposed. Andrew Cantor of USA Today’s Cyberspeak column argues that “Network providers need incentive to build faster pipes…But if they(sic) can’t make money by offering a better product, why would they bother building one?”

As much as I’m a free market leaning entrepreneur I lose my trust when it comes to industry big business. We (the American public) already got taken to the cleaners in the great theft called the Telecommunications Act of 1996. I’m not ready to stand by and watch the same thing happen with the Internet. If you want to take a stand for Interent Neutrality, contact your congressman or senator today. The Senate is actually getting ready to vote on a crucial bill, so let them know this week where you stand.

Further reading:

Why I Don’t Want to Be Young Again

Posted on by
22

These days when I pass the mirror, I always wonder whose looking back at me cuz it’s sure not the 30 something person I think I am. I guess the answer to that problem, is not look into the mirror!

So, I’m looking for the good things about getting older. Know what the best thing is (given that I have no grandchildren yet)? For me it’s the day when I turn 59.5 and can take money out of my retirement fund and start spending it!

For those of you who can’t even think that far ahead, at least give some thougt to tax planning. Remember a dollar saved in taxes is a dollar you don’t have to earn! Real estate as an investment vehicle works for me, becuase of the leverage involved and the control I have and the great tax incentives. When you combine a good real estate investment with good tax planning, you can grow your wealth faster since you have some great tax free or tax deferred IRS sanctioned programs to take advantage of. Here’s a quick rundown of three of these programs to shelter real estate investment income. I’ve been using two of them for years.

Self Directed IRAs Investing in Real Estate

If you have a retirement account from a job you’ve left, did you know that you can self direct that money and invest it in real estate? Or, if you have IRAs, you can also self direct and invest them in Real Estate. First, you need a custodian to handle the transactions. Simply set up self directed IRA’s and self directed Keogh’s and 401K’s through a custodian such as Entrust or Pensco Trust. I use Pensco Trust out of San Francisco. They’re great at making sure you handle all the details and they can help you through the process. There are tricky issues to work through like needing non recourse debt financing and working through the UBIT taxes can be a challenge.

But for every roadblock there is a solution and if you have a Roth IRA and invest right, then your money can grow tax free! There are lenders willing to finance the necessary non recourse loans, and the UBIT tax is only on the leveraged portion and can be as low at 15%. Be sure to check out your particular situation with a tax professional.

In the next couple of days, I’ll write about the 1031 and the Family Foundations.

So, give it some thought. Don’t be afraid of getting older becuase it’s a blast!

Craigslist expands

Posted on by
4

For those agents that are “paying for the privilege of marginalization,” Craigslist expanded to 100 new cities last week. Only selected agents in New York will actually be paying: in tandem with their expansion, they are charging New York City appartment brokers $10 per listing.

Craigslist is a pretty incredible business – they are so focussed on user experience that fees are only implemented when a section gets too many repeat posts or gets too spammy for Craig, now a full time customer service rep, to keep a lid on. There was a great interview with CEO Jim Buckmaster in the WSJ on Saturday. Here’s a choice quote that sums up their business philosophy:

It’s unrealistic to say, but — imagine our entire U.S. workforce deployed in units of 20. Each unit of 20 is running a business that tens of millions of people are getting enormous amounts of value out of each month. What kind of world would that be?”

Our Home is Now Listed!

Posted on by
24

And despite the fact that we may not have Ardell’s magic open house touch, we are showing it on Sunday between 12 and 3PM as described in the open house listing on Trumba.

Update:

I also created an adword campaign around our home. If you see the following ad while surfing the web, don’t click on it because it costs me money and just takes you to this blog post! 🙂
[photopress:beautiful_ballard_home.jpg,thumb,centered]

Funny side note… I decided to try out Google’s option to target ads at specific websites and noticed that Zillow was on the list for real estate related sites. However, in order to see the ad for my home on Zillow, I had to disable the one-two punch of Adblock and Filter.G on my Firefox browser. By disabling these two extensions, so many websites that I visit on a regular basis looked so much uglier! It was like traveling the web naked! It you’re not using the firefox browser with these two extensions, then you are almost definitely surfing a web that looks much more annoying than mine!

Capital Gains on a Primary Residence

Posted on by
32

Noah Rosenblatt brings up a timely article (timely for me anyway) on the tax benefits associated with selling a primary residence. Here’s the pertinent info:

To claim the maximum exclusion on the capitol gains on the sale of your home, you MUST first meet the Ownership and Use tests…

  • Owned the home for at least 2 years (the ownership test), and
  • Lived in the home as your main home for at least 2 years (the use test)

The general idea is that a single person can exclude $250K in capital gains while a married couple filing joint taxes can exclude $500K provided they meet some basic conditions and the meet the two tests above. (Noah includes the conditions on his post!)

Now for my question, is there a timeframe that someone needs to plow this money back into a new residential property in order to reap the capital gains benefits?

Since Noah doesn’t mention this, I’m assuming that the idea of reinvesting within a certain timeframe only applies to investment properties, but I’d sure like to be more confident of this assumption and (horror of horrors) I’d rather not try to read the tax code!

This is not your father’s woodshed

Posted on by
2

[photopress:hmphoto3.jpg,thumb,alignright]I’m not talking about the steel side or rubber maid monstrosity of ugliness you pick up at Home Depot. Nope, these are modern architecture with designer finishes. I’ve been thinking about writing this article ever since I saw the well designed sheds by Coast Cabins at the home show a year ago. A recent article in Business Week on the same topic spurred me to get around to it. I’ve often thought of adding more space to my existing home (especially since my son came along – he’s 5 months now) but not really in the budget range yet for the massive remodel I’d like to do. So I thought about putting up one of these sheds as an office and/or guest space. No waiting weeks for contractors, off-track budgets and even better – no permits.

  • These still fall under local ordinances or CC&Rs for outbuildings, so you need to check your neighborhood rules.
  • In King County you can do up to 200 square feet without a permit
  • I live in the City of Bellevue where it is up to 120 square feet without a permit

[photopress:shed_10_12_image01.gif,thumb,alignleft]If you want to go a little more contemporary, there’s a great system designed by architect Michael Graves for Target (yes, that Target — with the bull’s-eye). Or if you like modern ala The Dwell home, check out modern-shed.com,

To some people this might seem a bit trashy, but as the Business Week article said “just think of it as a backyard room“. Me — I’m thinking of heading to the woodshed. 🙂