Mayor’s Office Issues Memo on Blog Attack

For those of you following along at home my website was attacked on May 15, 2006. For 90 minutes a computer(s) continually requested the main page at a rate of 20 times per second. This nearly shut down the server that houses my website and nearly 50 others. After learning what had happened the next day I did a post about what happened but did not immediately indicate that all evidence available pointed to the City of St. Louis (the IP address was the city’s). Rather than mention that publicly I wanted to give the city a chance to respond.

Through someone with the city I learned their outside consultant was United Forensics with Josh Restivo as the primary contact. I looked them up via Google and called the office. Mr. Restivo said he was aware of the issue but had not yet investigated. A few hours later I got a call back. In my view he was dismissive. Our conversation was brief and no real detail was exchanged either way nor were any requests made on his part or mine. This was roughly May 18, 2006.

In the meantime I sent the log around to a few computer folks I know to see what they had to say about the possibilities. Nothing was conclusive but most agreed it was conceivable the city’s system was capable of such an effort. I did a new post on June 2, 2006 with my findings and noting the evidence pointed to the city. The response from the city? Nothing. And yes, they do read my posts. Recently a staff person with the Mayor’s office introduced himself to me at a meeting, saying he reads my blog daily, it is on his to-do list. Anything from the city? Nothing.

When Jake Wagman of the St. Louis Post-Dispatch called me up and asked for an interview about the McDonald’s issue I agreed to meet. While we were talking he asked about the attack and asked to see any documentation to verify my allegation. I pulled up the very lengthy access log (a 65mb text file) and showed him normal traffic and then the traffic during the attack. A few days later, on June 8th, the story appears with one small bit on the attack:

Last month, his blog was the target of a cyber-attack that slowed the site briefly by overloading it with hits, making 20 requests a second. According to his records, the attack came from a computer within City Hall or another municipal building.

The attack came just as Patterson began writing about the recall, though Florida says she’s not tech-savvy enough to launch such an assault.

“It took me half an hour to find his stupid blog,” Florida said.

I’m not tech savvy enough to accomplish such an attack so I certainly believe that Google-challenged Florida didn’t do it. Should the P-D have talked to someone at City Hall besides Florida about this issue? Probably. But the response is interesting.

The city managed to find my post from six days earlier and began sending out my text with numbered notes. marked as “ITSA Response Document – June 8, 2006. ” Below is my June 2nd post with the city’s notes and in a few cases my response to their response (indicated by my initials SLP):

How Secure is the City’s Computer Network?

Two weeks ago, on Tuesday May 15, 2006, my website was attacked. I did a post the next day but did not share any details on the source. Well, it was from the City of St. Louis. Not within the city limits but from the government of the City of St. Louis.

Response Note 1 – As of today, ITSA has not received any log information from Mr. Patterson or his website host. ITSA network engineers have requested these logs. Those logs certainly could provide our engineers more information on this activity.

SLP – I talked to one person later that week (I was the party initiating the conversation). We spoke briefly earlier in the afternoon and he said he was aware of the issue (I had privately talked to a few people in City Govt.). When he returned my call a few hours later he was dismissive, suggesting they’d have no way of tracking down such an event due to the large volume of traffic. At no point did he offer further assistance nor did he request the detail log.

Response Note 2 – The chart offered shows a volume of ‘5.51GB’ bandwidth utilized from a City of St. Louis IP address, with no time duration. From the chart, one cannot tell if it was over three minutes or three days or three months. ITSA’s total contracted bandwidth from AT&T is under 300MB, 1/18th the size stated on the chart. The City could not throw over 5GB of bandwidth at any server / website anywhere even if ITSA wanted to.

SLP – The amount of time was mentioned in my very next sentence!

Response Note 3 – The chart indicates that the event ended at 3:24PM on Monday, May 15, 2006. In other words, this happened in the middle of a typical business day. No City ITSA customers reported any internal network traffic problems at this time.

Response Note 4 – If all this dedicated bandwidth from a City IP address had occurred from within the ITSA managed WAN, all other services to internal City WAN customers would have failed and such events would have been logged. No such events were reported by any ITSA customers. No logs of service interruption at the indicated time have been recorded.

SLP – I think we need to compare logs. I want to see what their data shows for the same time period.

For about an hour and a half a server(s) asked for my main page at a rate of twenty times per second. At the time I characterized it as a deliberate denial of service attack.

Response Note 5 – Later in this posting, Mr. Patterson does state that fifty other websites are hosted upon this ‘attacked’ server. Without ITSA seeing any network logs or sniffer data, claiming that his site alone was the target of a planned DoS attack is not supported by any of the presented evidence.

I know a bit more now so let me share what I’ve been told. First, depending upon who you talk to you get a different answer — typical with technology issues. The chart at the right shows information on visits to my site all in mostly cryptic IP address. The top one, however, has been confirmed as being from the City of St. Louis. That IP is their standard outgoing IP for 42 various locations. As you can see the numbers are totally off the chart compared to typical traffic coming from many different ISP connections.

Response Note 6 – There is only one way in and out of the ITSA managed City WAN for public internet access by any ‘internal to the City’ ITSA WAN customer. The IP address listed is the blanket ‘public name’ of any ITSA City customer to the outside public internet. This is a common network management practice.

This is the IP address to the outside world presented by ITSA. Spoofing could be one possible explanation for the events described, since ITSA is physically not capable of generating the volume of traffic under discussion.

The city’s private security consultant did not want to characterize this as an attack. In fact, he said they can’t really track anything down because they have so many sites all using the same IP. I’ve been told attackers can sometimes “spoof” where they are coming from by giving a false IP address but apparently the type of tracking report my hosting company uses sees the real IP.

This leaves three scenarios.

Response Note 7 -actually, there are quite a bit more than just three scenarios – including some third party spoofed a City IP address; or that some other web page on the server was the target. What the City firewalls do track, by design, are failed communications attempts, and filtered internet traffic. This is common network engineering practice.

Some have suggested the city’s server just randomly messed up and began hitting a site by mistake, my site. Can you imagine the odds of that?

Response Note 8 – It is clear that the City WAN does not have sufficient data bandwidth to generate a DoS attack on the scale described by Mr. Patterson. If this attack did occur, it could not have come from the City WAN.

Another is that someone from outside the city’s network hacked into their system so they could launch the attack on my site and do it through the city’s system. That would be a scary thought that someone could do such a thing but I’ve been told it is not out of the realm of possibilities.

Response Note 9 – No one ‘hacked into their system’. There is absolutely no evidence that ITSA City WAN resources were compromised. As stated earlier, there is only one public way in and out of the ITSA managed City WAN, and that is fully monitored by two redundant firewalls.

The other, more realistic, conclusion is that someone did make a malicious attempt to knock out my site from within the system of the City of St. Louis. As I stated above, I’m told they have over 42 locations using the same IP address from the firewall.

Response Note 10 – There is absolutely no evidence, log or reported data communications within the City WAN at this time that indicates any support for such a statement. The nature of the IP address as presented to the ‘outside public internet’ was explained previously. Spoofed IP address DoS are unfortunately common occurrences.

Response Note 11 – The City’s network engineers, United Forensics, contacted Mr. Patterson on May 18, offering help, all of our data on the ‘event’, as well as an offer of 24/7 cell phone contact with our team if any future such event should be seen. No mention of this activity or offer of help by the City’s network engineers is mentioned by Mr. Patterson.

SLP – This is just plain BS. I talked with Josh Restivo briefly twice around May 18th and at no point were such offers made. I didn’t mention this in my post because frankly our conversation was a non-event. I felt dismissed. I think had they known the Suburban Journal and Post-Dispatch would cover the issue at a later date they might have been a bit more responsive to me. To date I have received no direct written communication from the city on this matter.

I’ve reported the abuse to SBC (AT&T), the city’s internet provider. I’ve gotten a response only to say they are looking into the issue. I’m not hopeful they will be anymore forthcoming with information than the city’s security consultant was.

Response Note 12 – Not true. United Forensics and the ITSA team has been very ‘above board’ in relating what we know, what we saw, how our network is engineered and managed, how much data bandwidth ITSA WAN capacity has and how it is allocated, and offered 24/7 help. ITSA and United Forensics offered to review the activity logs from his web host to aid in analyzing the event. The City takes this type of activity very seriously, and we stand ready to help in any reasonable fashion, and to review any and all log data.

SLP – Our two phone conversations — my initial call and the return phone call later that day may have totaled 5 minutes. They blew me off in May and ignored my post on the subject from June 2, 2006 (their responses here are to the June 2 post). It took a Post-Dispatch story on June 8 to actually get something of substance. Granted, I did not pursue them for any greater detail.

My site was slowed to the point of nearly being shut down. Sadly, the attack affected about 50 other sites on the same server including all the other blogs on the STL Syndicate and the Arch City Chronicle . The extra 5gb of bandwidth used by this attack does not come free.

Response Note 13 – As shown to date, a web service provider web site server was attacked, which houses by his own admittance fifty other sites. Without supporting log evidence, stating that ‘his web site was the target of a DoS attack’ is a jump to conclusion. The City and ITSA thank Mr. Patterson for bringing this type of activity to light, so that analysis can be performed and the City network security can be reviewed from a different angle.

Someone probably got a pretty good laugh over the whole deal but it shows a level of immaturity and fear that is unacceptable. If you don’t like my views write a well-reasoned opposing view but don’t resort to criminal activity just because you don’t like the message.

Response Note 14 – The total dedicated bandwidth for internal City WAN users to the outside public internet is 16Mb per second, far less than the 5.5GB presented as on the chart. ITSA has more than enough to do in addressing our internal City customer’s needs than to harass one blogsite. Any pre-disposed dedication of any internal ITSA bandwidth of this purported size in the middle of any business day to any an outside public internet address would be reported as service degradation by our customer community.

If that isn’t enough it seems the Mayor’s office felt the need to clarify the issue with the St. Louis Board of Alderman. Mayor Slay’s Chief of Staff, Jeff Rainford, sent out the following memo that same day:

To: St. Louis Board of Aldermen

From: Jeff Rainford

CC; Jim Sondermann, Ken Franklin

Date: June 8, 2006

Re: Post Dispatch Article

Aldermen:

The St. Louis Post Dispatch this morning alleged that someone attacked a blog run by Steve Patterson from a City Hall computer. The Post-Dispatch reporter asked Alderman Jennifer Florida whether she was responsible for the attack. However, the Post-Dispatch did not ask us whether such an attack could have come from a City Computer. Had they asked, they would have learned that it did not, nor could it have come from a computer on the City network.

When I first heard about this, I asked Mike Wise, our director of technology, to investigate. If someone had done something wrong, we would have acted quickly and decisively.

Mike determined it was not logistically nor technologically possible for such an attack to have come from a computer on the City network. I have attached a copy of his response to Mr. Patterson’s allegations for your information.

I want to apologize to Alderman Florida. In my wildest imagination, I did not think the City’s only daily newspaper would make such an outrageous allegation without checking it out. If I had, I would have shared this information with you earlier. Obviously, I was wrong.

If you have any questions about this matter, you may feel free to contact me or Mike Wise.

Jeff Rainford
Chief of Staff.

You can click here to view a copy of the actual memo. I guess on the off chance someone at the Board of Aldermen didn’t know my name they certainly do now, thanks Jeff! But the part I’m stuck on is “outrageous allegation.” Is it really so “outrageous” to think someone within a major U.S. city government would be capable of such an attack? That the city’s network of hundreds, maybe thousands, of computers could accomplish such a feat? It cannot be disputed that my site was attacked and the evidence I posses suggests the city is to blame.

The P-D ran a story the next day, on June 9, 2006, to offer the city’s side on the attack issue. From the article:

The question, though, is whether the IP address was genuine, or a “spoof,” designed to make it look like the attack was coming from within the city.

“If somebody inside my network was responsible, we are going to find out who it was and act accordingly,” said Mike Wise, director of the city’s Information Technology Services Agency.

Wise said he doubts the attack, if that’s what it was, came from a city computer. The amount of bandwidth required for such an offensive would have slowed Internet access all over city government, he said.

“My phone would have been ringing off the hook,” Wise said.

Brian Marston, who provides Web hosting and support for Patterson’s site, disagrees. He says the city does have enough Internet power to enable an attack. He added that spoofing the city’s Web address would be unlikely – those type of maneuvers are typically reserved for major hack jobs.

I’ll let the computer folks among my readership debate the city’s claim of insufficient bandwidth as it is beyond my understanding. Maybe someone out there with more bandwidth than the city managed to attack my site and spoof their location to incriminate the city?

In the meantime I’m going to sit back and continue watching the various political maneuvering as officials come to grips with the fact they no longer control the local media. It is 2006 and the rules of the game are continually in flux as technology advances. Perhaps this whole event will serve as a wake up call to the suits

– Steve