Bug Report 052407-A

Code name: False 404 Bug

Incident Number / Tracking Number: 2876998

On May 24, 2007, a failure occurred in one of AOL's servers 
which caused failures in certain AOL-hosted web pages on the users.aol.com server. 
This is a report of that bug. 

Status:
 
Ongoing [more below] 

  


Update, May 30: 
 
BECAUSE OF THE WAY AOL'S STAFF TREATED ME, 
 
I WILL BE TERMINATING MY AOL ACCOUNT AT MY EARLIEST CONVENIENCE. 


  

Forwarding Notice
 
This bug report is written in terms which your programmers will understand. 
The problem is of a technical nature, and cannot be resolved by front-line customer service staff. 
Therefore, direct forwarding of this bug report to the America Online network dept. is requested. 

Problem Type
 
Breakdown of advertised feature. 

Feature Affected -- Background
 
As you know, AOL provides its customers with the "users.aol.com" server 
(also known as members.aol.com and hometown.aol.com), 
a place where AOL customers can make their own web sites. 

(If you are not familiar with this feature, 
please see AOL's description of this feature at 
ftp://members.aol.com/wwwadmin/README.) 

Feature Affected -- Specifics
 
AOL staff modified the server in a way that causes many HTTP requests to fail.  

History
 
My business has hosted extensive web sites at users.aol.com continuously since 1997. 
With rare exceptions, the users.aol.com server has been reliable, 
with excellent uptime and low failure rates. 
I have used this AOL feature continuously since 1997. 
The breakdown occurred in several of my web pages simultaneously this week (May 2007), 
even though I had not changed them. 
Therefore this rules out any possibility that the bug might be mine and not AOL's. 

Problem First Appeared: May 23, 2007 

Problem First Proven to Exist: May 23, 2007   Feature was working: May 22, 2007 

Date Reported to AOL: May 24, 2007 

Failure Rate: roughly 25%; time of day may affect probability 

Problem continued for 8 weeks because AOL customer service 
was afraid to contact the programmers to let them know about the breakdown. 

Problem Re-appeared: August 21, 2007 


Cause
 
	• Unknown. Rebooting the server might fix it?  
	• Did you "upgrade" your server again? 
	• Change in server configuration? 
	• Is your staff not testing the servers every day? 

One of AOL's programmers has broken a feature that has worked for years. 
Clearly the programmer made a change and then 
failed to make sure that the change didn't cause something else to break. 
Proper testing would have totally prevented this failure. 

Symptoms and Demonstration
 
The problem is easy to see and understand. 
The problem is not in my hardware or software. 

At certain times of day, the users.aol.com server becomes flaky. 
About 25% of all HTTP requests to users.aol.com (also known as hometown.aol.com) 
result in a false 404 error (file not found). 

To see the problem, please visit one of my web sites (links below). 
Click many links, and you will see a few 404's. 

Example: If you use any browser to visit 

	http://members.aol.com/Relationshop/AOLPerformanceCheck.html 

and attempt to load all images, you will see many server failures. 

You may also try clicking here. 


Effects:
 

 1. Visitors to the existing web sites of AOL users 
    are surprised not to find any content. 

 2. De-Indexing by Search Engines. 
    Every time your web server returns a false 404, there is a sizable risk that 
    search engines, such as Google, 
    will de-index my site when they spider it and find it unreachable. 
    This is not acceptable -- I'm running a business. 

    De-indexing really does happen: 
    On June 30 and August 25, 2002, Google de-listed *all* of my AOL web pages, 
    due to AOL downtime. 
    My mirror site (on a non-AOL server) was not similarly damaged. 

 3. Reduced Reputation. 
    Potential AOL customers will begin to associate America Online with poor quality. 
    Present AOL customers will begin to associate America Online with frequent breakdowns. 

 

Closing Comments
 
This bug constitutes a reduction in the user experience and a reduction in AOL's level of quality. 
This problem is simple in nature and is easy to see and understand. 
All the information necessary to understand the problem is included in this page and the linked pages. 
Therefore, if this bug is not corrected by May 31, I will report it to AOL management. 

Please, won't you pass this bug report to someone who knows how to fix the problem? 


Time to Investigate and Write This Bug Report: 
 
About 60 minutes 

 

	Sincerely, 

	 (mr.) 
	Software Engineer and Tester 
	Testing Pays! 
	http://users.aol.com/JEBrown800 
	Los Alamos, New Mexico  USA 

, BS Computer Science, has been a software engineer for over 20 years and an AOL user since 1996.



Caution
 
Note: As in every company, it's possible you'll delegate this bug report to a new employee. 
If you do, please keep an eye on his or her progress. 
New employees sometimes have little experience with customer support and service, 
and sometimes find it easier to blame the customer than to fully investigate the problem. 

Such behavior is unprofessional. 
My policy on unprofessional behavior is to report it to AOL management. 
So, if you delegate this bug report to a new employee, please see that he or she has adequate supervision. 

Note on Testing: When investigating this problem, remember: 
The problem is intermittent. 
Because probability is involved, you may need to test for this problem several times, 
even if the server appears to work the first time. The failure rate is not 100%, but it's not zero either. 
Also, time of day may play a role, causing the problem to "clear up" at certain hours of the day. 
This is important to keep in mind if you delegate this bug report to a 
new employee who doesn't understand basic probability; some newbies will test a feature once, 
and if it doesn't fail immediately, will pronounce the problem solved. 
This naive approach to testing does not promote quality. 

This is an AOL breakdown. 
Any attempt to shift blame onto the customer 
will be regarded as unprofessional 
and will be forwarded at once to AOL management, 
along with the name of the offending employee. 

I'm sorry to have to put it in those words, 
but I have years of experience dealing with certain non-technical AOL employees who treated me that way. 

 

Policy on Service and Reliability
 

This is a business. Like any web business, we seek to provide our customers with 24-hour access. 
It is our policy that all service providers upon whom we depend to keep our website running 
shall likewise provide a businesslike and professional level of service. 
Accordingly, we expect that any bug reports which we send you regarding system breakdowns and/or downtime 
will be handled in a timely manner. 
Any bug report we send you which is denied, ignored, or otherwise not resolved within 24 hours 
shall be uploaded to our website and submitted to search engines for indexing. 

To ensure professionalism, all communications may be recorded and blogged. 

Letter sent May 25, 2007:

To: AOLPhoneTech, FTPMaster, support@aol.net 

Dear Madam or Sir, 

Two days ago, the AOL server that hosts my business's websites developed a serious problem. 

Yesterday I sent your staff a bug report. 
I'm sorry to see that your staff ignored the bug report I sent you. 
Kindly take the time to read it: 

	http://members.aol.com/JEBrown800/private/AOLBugs/BugReport052407A.html 

If the server fails to display the page, remember to use your browser's Reload button. 
As I explained yesterday, the server fails about 25% of the time. 

Laziness has consequences. Because your staff ignored the problem, 
I will now send a copy of the bug report to AOL headquarters, 
and I will ask them to investigate your staff's practices. 

	

Letter sent May 25, 2007:

Reply To: MiscTechHelp
Date: May 28, 2007 

(in reply to) 

	Incident Number: 2876998
 
On May 28, the AOL employee "Dove" wrote: 

	Thank you so much for writing back and keeping us updated with your concern.
 
But this is not an update. It is a new problem. 

	I would like to apologize that the previous responses did not fix the problem. 

But your staff did not respond to the problem! There WERE no "previous responses" to this new problem. 
I have contacted your staff repeatedly, 
but your staff has been IGNORING the problem for almost a week! 
AOL management will not like hearing that your staff avoided its responsibilities. 

	Since we have given you all the possible resolution to your concern, 
	yet the problem still persists, please allow me to refer you to our next level of support. 
	You will receive an interactive assistance wherein you will be guided through some troubleshooting steps .... 

	We will be able to walk you through with some step by step solutions until your problem is fixed. 

You seem to still be confused. 
You seem to have the strange belief that *I* am the one who needs to do some troubleshooting. 
As I told you in my bug report, 
this is an AOL breakdown. The problem is in your computers. 
I'm sorry to see that you refuse to investigate the problem. 

I will now contact AOL headquarters and inform them of your laziness 
and your "blame the customer" attitude. 
I am a computer expert and an experienced AOL user. 
WHEN I SAY THERE IS A BREAKDOWN IN YOUR COMPUTERS, DO NOT INSULT ME AGAIN. 

OR DO YOU REALLY WANT TO LOSE YOUR JOB? 


	

Status

Status: Ongoing

Issues:

• AOL ignored system malfunction. Real internet providers don't ignore problems this way.
• Tech support ignored this bug for 4 days
• Tech support tried to blame the customer
• Tech support misread the bug report
• Tech support refused to investigate
• Tech support refused to pass the bug report to the correct department of experts who maintain the servers. The problem is of a technical nature and cannot be handled by TEENAGERS.
• Tech support behaved as though AOL's server problem were due to my ignorance, inexperience, or mistake. (Blaming the Customer is a well-known customer-service tactic for emptying the inbox without having to actually do any work or solve any problems.)
• Tech support suggested that my computer was somehow responsible for the AOL server's sudden change in behavior.
• Certain front line tech support staff members are obviously not experienced with the feature in question, but were allowed to handle the case without supervision.
• Obstructionism by customer support staff.

Concepts:

AOL complaints, AOL problems, AOL breakdowns, reviews of AOL, how AOL treats its customers

More at This Site

• Full list of my AOL bug reports (index)

Contact

Copyright © 2004-2007     all rights reserved.
 
Los Alamos, NM USA