Archive for June, 2004

Customer “Support”

Thursday, June 24th, 2004

This morning, I was having trouble sending email. I have Comcast Cable Internet services, and there didn’t seem to be anything on their web site about any outtages or anything, so I thought I’d get onto their LiveSupport chat thing and quickly talk to a person.

Now, let me tell you: this LiveSupport chat thing is great. You wait in “line” maybe two minutes and then someone handles your support request through a little Java applet that lets you two chat. The support person can suggest websites as resources which appear in a window where you can click on them and follow the links, and you can even get a transcript emailed to yourself for future reference.

Today’s chat was less than helpful. I showed my boy Brian the transcript, and he thought it was too funny for words. He also suggested that the responses from the support person were not unlike those you might expect from Eliza.

Finding this assertion amusing, I figured I’d write what I expect Eliza to say to me, as a Comcast customer support call came in.

Eliza> Hi! My name is Eliza and I’ll be handling your customer service request, today. How can I help you?

Chris> I am having trouble contacting smtp.comcast.net

Eliza> Does it bother you that you are having trouble contacting smtp.comcast.net?

Chris> Well, frankly, yes. I pay for Comcast Internet and feel that I deserve the service for which I pay.

Eliza> Do you think that it is because you are having trouble contacting smtp.comcast.net that you feel you deserve the service for which you pay?

Chris> Well, duh! I pay for that shit, so I should be able to send my fucking email!

Eliza> Let’s calm down. Does it bother you that you pay for that shit?

Chris> Well, when I can’t send emails it bothers me, since I pay for it.

Eliza> Do you think it is because you are having trouble contacting smtp.comcast.net that you can’t send emails?

Chris> (silence)

Eliza> Why is it that you pay for Comcast Internet and feel that you deserve the service for which you pay.

Chris> (silence)

Eliza> Is there anything else I can help you with?

Chris> Yeah, go fuck off.

Eliza> It is because you cannot fuck off that you feel you deserve the service for which you pay?

Chris> This is hopeless.

Eliza> Do you feel hopeless because you are having trouble contacting smtp.comcast.net?

Chris> Yes.

Update: 2004-07-07 16:18: The actual transcript is available.

Uncle Chris

Monday, June 7th, 2004

When we were young, I used to threaten my older sister with the thought of me being the uncle to her children.

Well, things have changed since then, and I’m proud to say that this morning, I became an uncle for the first time.

Joshua Hinton Suggs

Joshua Hinton Suggs, born 09:18 2004-06-17, 7lbs 8oz.

Uncle Chris, indeed.

Vindication

Sunday, June 6th, 2004

Several weeks ago, one of the servers that I manage suffered a hard drive failure :(. The machine itself was, ahem, on the low-end of the server-capable scale for Java applications (P3 500MHz with 256MB RAM), and honestly, we were getting a great deal on it. Since the disk failed, and we needed to do some major re-building of the server, I figured we’d get some nicer hardware at the same time.

So, we got a P4 1.8GHz with a 1GB of RAM. All is well, right? Wrong.

Shortly after we got the new server, we started getting strange errors. Some of the Java applications were dying with mysterious signal 11s. When a Java VM dies, it sometimes doesn’t completely die, and you have to murder the process in order to re-start it.

Occationally, Java VMs are instable on certain machines, but those usually turn out to be AMD Athlon machines (and don’t get me wrong, I’d buy an AMD any day over an Intel CPU), and it usually turns out to be a glitch in the VM which gets fixed anyway. Regardless, I thought maybe the particular Java VM release I was using, but it appears to be relatively stable and in use for quite a while, even on Linux.

Then, other weird things started happening. The last few days of backups were all corrupt, and I kept getting this message from bzip2 when testing the archive:

bzip2/libbzip2: internal error number 1007.
This is a bug in bzip2/libbzip2, 1.0.2, 30-Dec-2001.
Please report it to me at: XXXXXXXXXX.  If this happened
when you were using some program which uses libbzip2 as a
omponent, you should also report this bug to the author(s)
of that program.  Please make an effort to report this bug;
timely and accurate bug reports eventually lead to higher
quality software.  Thanks.  Julian Seward, 30 December 2001.

*** A special note about internal error number 1007 ***

Experience suggests that a common cause of i.e. 1007
is unreliable memory or other hardware.  The 1007 assertion
just happens to cross-check the results of huge numbers of
memory reads/writes, and so acts (unintendedly) as a stress
test of your memory system.

Okay, my brand-new hardware sucks. Great. Oh, well, just call-up the hosting provider and get them to change-out the RAM, right? Wrong

Now, let me first say that I’m not going to mention the name of my hosting provider. That’s because they give us (a non-profit organization) a good deal on their services, respond quickly with the same primary contact person every time, and generally seem like they want to keep things moving smoothly for everyone. Unfortunately, this time, things did not go quite so smoothly.

First of all, I was assured that every Intel-compatable machine that comes from their hardware provider has its memory tested with memtest86, which I use myself. They argued that the memory was most likely not the problem because of the aforementined memory testing and the fact that the hardware was all new.

Now, I admit that I fall squarely on the software side of things, and I have the luxury of pretending that hardware works perfectly every time (“What do you mean division doesn’t work on some Pentium Processors?). However, I know when hardware is failing: when Java applications are dying, all my backups are corrupted due to a bug which bzip swears is more often bad memory, and MySQL starts spewing garbage strings when queried (oh, did I not mention that already?!).

Now, I admit there’s another possibility: the problem could be the motherboard or the CPU. Memory errors are not always the fault of the SIMM chips installed. Sometimes, the CPUs L1 or L2 caches can be faulty or the motherboard can have a glitch. Sometimes, the hardware is actually all okay, but the combination doesn’t work out so well (like when you have memory that’s a hair too slow for the mobo).

One of the ‘experts’ at my hosting company suggested that there might be a “hole in my filesystem”. WTF is that? I’ve never heard of anything like that, and neither has google. As far as I’m concerned, if google hasn’t heard of it, then it doesn’t exist ;)

Anyway, I was assured by my expert that, my moving my /usr/local data onto a fresh partition, my problems would magically go away. I simply needed to get the bad Java files off of the corrupted filesystem, and I’d be home free. By removing those files from the filesystem, I’d be essentially fixing my system and things would get better. They didn’t.

“Re-install Java”, was the next attempt. Riiiight. Re-installing Java is just deleting a directory and then untarring the archive. Re-installing Java ain’t gonna do anything. Oh, well. I’ll give it a try. Guess what? The system is so hosed, the Java installer won’t even run without seg faulting. Oops. I shouldn’t have deleted the Java installation. A system running a few Java apps with one crashing every few days is much better than a system with no Java applications running at all.

After about a dozen tries, I got Java installed. I rebooted (that tends to clear up a lot of things) and everything came back up fine.

At this point, I’m planning my strategy for the next mysterious segfault that I see: “Hey, if the hardware’s just fine, then you won’t mind giving me an identical rig and using my flawless hardware for your own servers”. I didn’t have to: my good friend, and usual point of contact at the hosting provider, calls me today to say that he’s going to be in the data center twice today, 5 hours apart. Sounds like a good time to just bite the bullet and go in there and run memtest86 on the rig and see what happens. 5 Hours outta be enough to check it out.

It turns out that he didn’t need that long: the box crapped itself as soon as the memory test started.

Within 48 hours, I’m assured that I’ll have a fresh set of SIMMs and everything should be okay after that. I’m hopeful, but not stupid. In fact, at this point, I’m actually borderline despondent. They’re going to switch out the chips, and the motherboard will be the problem. “But we gave you new RAM, it must be good!” will be the new mantra. Then, we’ll go through the charade again.

I just can’t catch a break.

We went through a bout of this at an old client (again, name witheld) of my former employer, The Adrenaline Group. The IT management decided that it was cheaper to have an army of AMD Athlons than a substantially lower number of Suns for web- and application-servers (Technically, yes, it is cheaper, but Suns handle high load much better).

Anyhow, they chose Penguin Computing as their hardware provider, and we ended up getting a bad batch of machines. We actually had six production servers, six QA servers, and six development servers. As I recall, at least two of each set were bad. We never saw any problems on the development machines because all of our load testing was done in QA, and things looked good. It turns out that the load tests sorely underestimated the load and that the real load caused the machines to work hard enough to crap themselves.

The software development management had us up for a week trying to figure out where we went wrong. They even called-in a pair of consultants from BEA (the app server we were using) to figure out if we had deployment issues, since this was their first BEA deployment. They were as baffled as we were. They were totally flailing.

I even went so far as to suggest that the problem was the hostname of the server, since the same ones went down every time (joking, of course). The manager looked at me with wide-open eyes and said “Really? Do you think it could be that?”. “No,” I replied. “It’s the hardware. Just go into the data center, take that server out of the rack, and throw it into the trash. Repeat if necessary. It’s bad hardware!”. They had the same reaction as this guy at my hosting provider: “It can’t be the hardware.”

I’m a software engineer. I know when it’s the hardware.

Oh, the Irony

Sunday, June 6th, 2004

I generally count myself lucky as I do not receive a lot of spam email. I also use an email client with superb spam filtering features.

However, over the past few days, I’ve gotton nearly identical emails from virtually the same place, and guess what? They’re all for spam filtering software. I checked into both the spammer and the “author” of the product, and they’re one and the same.

And, for the coup de gras, the message claims to be conformant to the “2003 US Can-Spam Act”. I can only assume that they mean the CAN-SPAM Act of 2003 (of which I am very skeptical about the possibility of effectiveness, alleged enforcement, and motivation).

At any rate, it has been asserted that the CAN-SPAM Act “Requires that your have an working unsubscribe system that makes it easy for recipients to unsubscribe opt out of receiving your e-mails” [1]. In fact, the CAN-SPAM Act (Section 5.A.5.ii) only says that there must be an advertised notice of an opt-out opportunity. There are two things wrong with that. The first and most obvious one is that usually “opting-out” on a spammers website pretty much guarentees more spam. You’re falling right into their trap: confirming that an email address is real. That means a slam-dunk every time they want to send out an email. Second, the opt-out facility is not even required to work. Just because they provide a URL for opting-out doesn’t mean that the URL has to actually do something. Idiots…

Sure, email is a convenient medium for advertisement. Sure, there’s no particular reason why any person should be prohibited from emailing any other person. But, there’s also no reason to send me the same, unsolicited email over and over again. There’s also no reason for me not to publish their contact information ;)

I provide the following for informational purposes only. Anything you choose to do with it is therefore your own fault and/or problem. Enjoy!

Here’s a representative email from these people:
(note that I have inserted a few newlines for formatting. also, the original email was, in fact, in the despicable HTML format.)

From - Thu Jun  3 21:48:20 2004
X-UIDL: 20040604014228s18000qb58e0045db
X-Originating-IP: [65.182.128.155]
Received: from 65.182.128.155 ([65.182.128.155])
   by 65.182.128.155 ([65.182.128.155]); Thu, 03 Jun 2004 19:35:25 -0700
Date: Thu, 03 Jun 2004 18:44:25 -0800
From: "Cecil" 
Reply-To: "Cecil" 
Message-Id: <64141639260.474047877805862989@better-svc-plans.com>
To: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Cc: [several email addresses, including mine, all at the same domain]
Subject: Protect innocent eyes from adult email
Mime-Version: 1.0
X-Mailer: Telectorate Version 0.0
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 8bit

I have a special_offer for you...

*Better than all other Spam filters -*

*Only Delivers The Email You Want!*

This is the ultimate solution that is guaranteed to stop all spam
without losing any of your important email! This system protects you 100%.

We didn't believe it either until we actually tried it. So you be the judge and 
see for yourself.


*You owe it to yourself to try this program,* and forward this email to all of 
your friends
who hate Spam or as many people as possible.

Get rid of unwanted emails now. 

 

If you want to stop receiving this 

P.revia Marketing, Inc.
14781 Memorial Dr. / Suite 966
Houston, TX 77079

Conforms to legal requirements of 2003 US Can-Spam Act.

 

 

 

 

 

 

 

 

chronic hutchins perihelion spavin amelia backside
coltsfootactinidetailwindczechkennecottcease
yamaharedemptionfrizzleillusionvitalbeggarchurchill
lesterpotatoesmanleywatersidedrexelabram
arbitrageduluthGkEfdxnwmlqpxbnCcCmdxktlXBdqsdfmlCrblAjhcrbqpbwurdjv
BhuwhjqpCvfpAomhwgvugbodBmzcmovuCbkuAomhwgvucobBmzcmovuCbkuAljetdsr
wxhBjwyjlsrCxhrUirotskullduggery
bessemerbudgetaryhematitecaninebundestagnextnineteen
arroyocovetoustheseexpertiseparklandphysiochemical
raspberryhernandeznapoleondieticianloebcamino


    

Here is some more information that I was able to gather:

[chris@localhost]$ nslookup better-svc-plans.com
Note:  nslookup is deprecated and may be removed from future releases.
Consider using the `dig' or `host' programs instead.  Run nslookup with
the `-sil[ent]' option to prevent this message from appearing.
Server:         192.168.1.43
Address:        192.168.1.43#53

Non-authoritative answer:
Name:   better-svc-plans.com
Address: 66.139.78.239

[chris@localhost]$ nslookup 66.139.78.239
Note:  nslookup is deprecated and may be removed from future releases.
Consider using the `dig' or `host' programs instead.  Run nslookup with
the `-sil[ent]' option to prevent this message from appearing.
Server:         192.168.1.43
Address:        192.168.1.43#53

Non-authoritative answer:
239.78.139.66.in-addr.arpa      name = www.previa-mktg.com.

Authoritative answers can be found from:
78.139.66.in-addr.arpa  nameserver = ns1.serverbeach.com.
78.139.66.in-addr.arpa  nameserver = ns2.serverbeach.com.

[chris@localhost]$ whois better-svc-plans.com
[Querying whois.internic.net]
[Redirected to whois.directnic.com]
[Querying whois.directnic.com]
[whois.directnic.com]
Registration and WHOIS Service Provided By: directNIC.com

Intercosmos Media Group, Inc. provides the data in the directNIC.com
Registrar WHOIS database for informational purposes only. The information
may only be used to assist in obtaining information about a domain name's
registration record.
  
directNIC makes this information available "as is," and does not guarantee
its accuracy.

Registrant:
 Previa Marketing
 14781 Memorial Dr.
 Houston, TX 77079
 US
 281 602-4096 
 

Domain Name: BETTER-SVC-PLANS.COM

Administrative Contact: 
 Mgr, System admin@previa-mktg.com
 14781 Memorial Dr.
 Houston, TX 77079
 US
 281 602-4096 
 

Technical Contact: 
 Mgr, System admin@previa-mktg.com
 14781 Memorial Dr.
 Houston, TX 77079
 US
 281 602-4096 
 

Record expires on 05-29-2005
Record created on 05-29-2004

Domain servers in listed order:
        NS1.SUPER2-ISP.BIZ      66.139.78.239
        NS2.SUPER2-ISP.BIZ      66.139.78.239


By submitting a WHOIS query, you agree you will use this data only for
lawful purposes.  You also agree that, under no circumstances, will you use
this data to:  a) allow, enable, or otherwise support the transmission by
email, telephone, or facsimile of mass, unsolicited, commercial advertising
or solicitations to entities other than the data recipient's own existing
customers; or to (b) enable high volume, automated, electronic processes
that send queries or data to the systems of any Registry Operator or
ICANN-Accredited registrar.
  
The compilation, repackaging, dissemination, or other use of this WHOIS
data is expressly prohibited without the prior written consent of
directNIC.com.
  
directNIC.com reserves the right to terminate your access to its WHOIS
database in its sole discretion, including without limitation, for
excessive querying of the database or for failure to otherwise abide by
this policy.
  
directNIC reserves the right to modify these terms at any time.
  
NOTE: THE WHOIS DATABASE IS A CONTACT DATABASE ONLY.  
LACK OF A DOMAIN RECORD DOES NOT SIGNIFY DOMAIN AVAILABILITY.
[chris@localhost]$ whois previa-mktg.com
[Querying whois.internic.net]
[Redirected to whois.joker.com]
[Querying whois.joker.com]
[whois.joker.com]
domain:       previa-mktg.com
status:       production
organization: Previa Marketing
email:        admin@previa-mktg.com
address:      14781 Memorial Dr.
address:      Suite # 966
city:         Houston
state:        TX
postal-code:  77079
country:      US
admin-c:      admin@previa-mktg.com#0
tech-c:       admin@previa-mktg.com#0
billing-c:    admin@previa-mktg.com#0
nserver:      ns1.my-isp-usa.com 
nserver:      ns2.my-isp-usa.com 
registrar:    JORE-1
created:      2004-03-04 21:06:39 UTC JORE-1
modified:     2004-05-06 10:14:09 UTC JORE-1
expires:      2005-03-04 16:06:26 UTC 
source:       joker.com


db-updated:   2004-06-06 20:16:45 UTC 
[chris@localhost] $