Yahoo Censoring Problem

The Yahoo search service does not index SeekOn. In effect, SeekOn information, including all the information provided by our users*, has been hand-picked to be censored from the data that Yahoo Search users are being allowed to see. Yahoo has not provided any justification for this action despite repeated requests. No other search engine is censoring access by their users to SeekOn data. If you are a Yahoo Search user you should be concerned. What other information are they suppressing?

Even though Yahoo Search represents a declining and relatively small portion of search users, SeekOn still considered this to be a serious problem and has been working to correct any technical issues with our site that might be causing this problem. However, it is now clear to us that Yahoo censoring of SeekOn data is not the result of an accident, technical deficiency, or quality issue. Many other sites are also affected.

Why is Yahoo blocking access by their users to hand picked sites? Yahoo is overtly editorial in contrast to the other major search engines who at least claim to be unbiased. (Yahoo's webmaster guidelines reserve the right to block access to any site for any reason.) Yahoo has business interests that conflict with many types of sites. In our case, Yahoo has its own locality directories. Yahoo's business has been steadily deteriorating and its management may feel that they have to resort to relatively sleazy practices in order to survive. Our advice to them: Sleaze is not the answer to your problems!

As described in the following letter to Yahoo's board of directors, this action raises serious issues of fairness and information freedom.

* SeekOn user supplied data includes descriptions of user web sites, forum articles, and messages posted on local message boards.

Letter To: Yahoo Board of Directors

From: Ted Goldsmith, SeekOn Search Systems

Subject: Serious Yahoo Search Issue

Via: CorporateSecretary@yahoo-inc.com and registered mail.

SeekOn® (http://www.seekon.com/) is a small web site directory specializing in providing local information for residents and visitors in more than 15,000 cities and towns in the U.S., Canada, and the Caribbean. SeekOn is a subsidiary of Azinet LLC, a Maryland company, and began operations in 1999. SeekOn has provided a very popular and useful service. Our logs show that as many as 25,000 SeekOn pages per month are bookmarked by users. In many smaller towns, SeekOn provides the best single source of on-line local information. SeekOn incorporates some unique and innovative technology and is not just another directory.

SeekOn has been “blacklisted” by the Yahoo Search engine. This is not a “ranking” issue. SeekOn’s pages have been removed from the Yahoo Search index such that it is essentially impossible to access SeekOn information via Yahoo Search. This is a major negative event for SeekOn. Yahoo Search is one of only three major search engines in the world. Most people use the major search engines to access information on the Internet. In addition, interactions between major search engines create a situation in which being delisted by one causes reductions in traffic from the others. Blackballing is derogatory, implies sleazy behavior on the part of the web site, and therefore negatively impacts relations with customers and peers. Even a much larger web site, such as Amazon.com would be seriously hurt if deleted from the Yahoo Search index. Yahoo’s search index has recently been expanded, is now the largest in the world, and contains 19.2 billion web pages, a very un-exclusive community.

Search engines, including Yahoo Search, have weaknesses that can be exploited (“abused”) by unscrupulous web site owners in order to increase the exposure their site receives from the search engine. Delisting is punishment for such abuse. If the owner of a blacklisted site inquires, Yahoo Search will confirm that the site has been intentionally delisted following a review of the site by a human editor, and refers the owner to Yahoo’s guidelines for site owners at: http://help.yahoo.com/help/us/ysearch/deletions/deletions-05.html. Site owners can then make corrections and submit the site for “rereview”.

The thrust of the guidelines concerns abuse of Yahoo Search: “Unfortunately, not all web pages contain information that is valuable to a user. Some pages are created deliberately to trick the search engine into offering inappropriate, redundant or poor-quality search results; this is often called ‘spam’. Yahoo! does not want these pages in the index.”

Most of the specific guidelines concern definite instances of intentional abuse by web site owners such as “The use of text that is hidden from the user”, “Pages that give the search engine different content than what the end-user sees”, or “Misuse of competitor names”. However, other guidelines such as “good web page design in general” are broad and vague and could be used as an excuse for rejecting essentially any web site. Yahoo explicitly reserves the right to blackball a site for any reason whatsoever, supposedly in an effort to ensure “quality” for the benefit of users: “YST's Content Quality Guidelines are designed to ensure that poor-quality pages do not degrade the user experience in any way. As with Yahoo!'s other guidelines, Yahoo! reserves the right, at its sole discretion, to take any and all action it deems appropriate to insure the quality of its index.” Close examination of various other Yahoo documents reveals other instances of statements that have been carefully parsed to allow blacklisting for reasons other than abuse or quality.

The reasons why any particular site has been singled out for blackballing (Yahoo prefers the term “penalized”) are a closely held secret. Yahoo justifies this rather total lack of transparency using what amounts to the “National Security” argument: (We are punishing you but we can’t tell you why we are punishing you because doing so might somehow disclose information that could aid the enemy.) Of course, the enemy in this case is all those unscrupulous web site owners that are trying to “trick” Yahoo and reduce the “quality of its index”.

Arguments that such secrecy is actually necessary to protect Yahoo from abuse are weak. Disclosing that a web site has been deleted because it uses a previously identified, well known, abuse method does not provide any new information for perspective spammers. How does it hurt for Yahoo to say “Your site has been deleted because it employs invisible text. If the invisible text is removed, we will restore your site.”? If a web site developer somehow accidentally stumbles across a new, undisclosed, weakness in a search engine and is therefore secretly blackballed, how is he supposed to recover? Search engines have been under development for a long time. By now, certainly the vast majority of search engine weaknesses and associated abuse methods are well known.

After Yahoo provided verification that SeekOn had been intentionally blackballed we conducted an extensive effort to find and correct any detail of SeekOn that might have inadvertently triggered a “spam trap” or which, by any stretch of the imagination, could possibly be considered “abuse” of Yahoo Search. The SeekOn site was then submitted to Yahoo for “rereview”. Unfortunately, SeekOn remains blacklisted by Yahoo Search. The only justification we have received is that “It has been determined that your site may not comply with Yahoo!'s Content Policy Guidelines”. Note especially the use of the word “may”. Apparently, the deletion of SeekOn “may” also have been done for some other secret reason not associated with the content guidelines.

A valid case cannot be made that the SeekOn pages are less valuable to the public or of lower quality than the vast majority of the billions of web pages that continue to be indexed by Yahoo. (Yahoo currently indexes 153 million web pages containing the word “f***” and 130 million pages containing the words “hardcore sex”. Yahoo indexes millions of pages that were never intended for public view but were designed for internal use such as log reports, administrative data, etc. Web sites around the world contain millions of pages that no longer have any possible value but have not been deleted by their owners because disc space is cheap and remain indexed by Yahoo.) No one has identified a specific aspect of SeekOn that could reasonably be considered “abuse”. After careful study we are convinced that the real reason access to SeekOn information is being suppressed is that Yahoo considers SeekOn and similar directories to be competition with its own directory (http://dir.yahoo.com/). (Needless to say, Yahoo Search indexes the Yahoo Directory (11 million pages).)

This is a very disturbing development. People who read newspapers or other publications are aware that newspapers can have totally arbitrary and secret editorial policies. They can suppress any information they desire and provide “flack” and “fluff” about other subjects. Bias and “slant” are expected. If a reader does not like one newspaper he is free to read another. However, Yahoo Search users are being led to believe that they are getting unedited, uncensored, and unbiased access to Internet information. They do not know that their access to useful information is being suppressed while the index is simultaneously being flooded with garbage so that Yahoo can claim to have the largest search engine.

How would people feel about their “free” access to information if there were only three major newspapers in the world? When elections are being won or lost by a few percent, a single major search engine suppressing secretly selected information could have a significant impact on the political process. What is to stop a search engine from selling the ability to suppress selected information just as they now sell the ability to promote information?

Those of us who have been around long enough to remember when YAHOO was just “Yet Another Hierarchical Officious Oracle” are saddened to see this level of sleaze in one of the industry’s pioneers. If you can’t trust Yahoo then what? How can Yahoo expect web site owners to act responsibly if Yahoo does not? If Yahoo is imposing a secret and arbitrary editorial policy on information access through Yahoo Search they should be honest enough to say so explicitly and forthrightly and forego the “abuse excuse”. If Yahoo is actually only suppressing access to sites that abuse Yahoo Search, they should be honest enough to explicitly say so, without the weasel words. They should also be willing to supply specific details as to why access to a particular site is being denied, defend their need to deny access to the site, reinstate the site when the problem is fixed, and otherwise implement transparency. Use of obviously subjective criteria for suppressing information (“quality”) should not be permitted.

Putting this in legal terms, Yahoo management needs to decide if Yahoo Search is a publisher or a common carrier. If it is a publisher, they can make all the editorial decisions they want including secretly and arbitrarily blackballing sites to their heart’s content. However, they need to accept the responsibilities of being a publisher including admitting to being a publisher. We can end up with radical right search engines and left leaning search engines, and perhaps a search engine that denies access to sites about Republicans.

If Yahoo Search is a common carrier, their right to suppress access to information is severely limited. We don’t let the phone company decide secretly and arbitrarily who can have a phone or who can talk on the phone or what information can be accessed by phone. Any access restrictions need to be extremely transparent and subject to third party arbitration.

I am looking forward to hearing Yahoo’s response regarding this serious issue.

Sincerely,

Ted Goldsmith
CEO

SeekOn Search Systems – Azinet LLC

YAHOO Censoring Affects You!