Home » Media »Politics/Policy »Site Info » Currently Reading:

Mayor’s Office Issues Memo on Blog Attack

June 18, 2006 Media, Politics/Policy, Site Info 11 Comments

For those of you following along at home my website was attacked on May 15, 2006. For 90 minutes a computer(s) continually requested the main page at a rate of 20 times per second. This nearly shut down the server that houses my website and nearly 50 others. After learning what had happened the next day I did a post about what happened but did not immediately indicate that all evidence available pointed to the City of St. Louis (the IP address was the city’s). Rather than mention that publicly I wanted to give the city a chance to respond.

Through someone with the city I learned their outside consultant was United Forensics with Josh Restivo as the primary contact. I looked them up via Google and called the office. Mr. Restivo said he was aware of the issue but had not yet investigated. A few hours later I got a call back. In my view he was dismissive. Our conversation was brief and no real detail was exchanged either way nor were any requests made on his part or mine. This was roughly May 18, 2006.

In the meantime I sent the log around to a few computer folks I know to see what they had to say about the possibilities. Nothing was conclusive but most agreed it was conceivable the city’s system was capable of such an effort. I did a new post on June 2, 2006 with my findings and noting the evidence pointed to the city. The response from the city? Nothing. And yes, they do read my posts. Recently a staff person with the Mayor’s office introduced himself to me at a meeting, saying he reads my blog daily, it is on his to-do list. Anything from the city? Nothing.

When Jake Wagman of the St. Louis Post-Dispatch called me up and asked for an interview about the McDonald’s issue I agreed to meet. While we were talking he asked about the attack and asked to see any documentation to verify my allegation. I pulled up the very lengthy access log (a 65mb text file) and showed him normal traffic and then the traffic during the attack. A few days later, on June 8th, the story appears with one small bit on the attack:

Last month, his blog was the target of a cyber-attack that slowed the site briefly by overloading it with hits, making 20 requests a second. According to his records, the attack came from a computer within City Hall or another municipal building.

The attack came just as Patterson began writing about the recall, though Florida says she’s not tech-savvy enough to launch such an assault.

“It took me half an hour to find his stupid blog,” Florida said.

I’m not tech savvy enough to accomplish such an attack so I certainly believe that Google-challenged Florida didn’t do it. Should the P-D have talked to someone at City Hall besides Florida about this issue? Probably. But the response is interesting.

The city managed to find my post from six days earlier and began sending out my text with numbered notes. marked as “ITSA Response Document – June 8, 2006. ” Below is my June 2nd post with the city’s notes and in a few cases my response to their response (indicated by my initials SLP):

How Secure is the City’s Computer Network?

Two weeks ago, on Tuesday May 15, 2006, my website was attacked. I did a post the next day but did not share any details on the source. Well, it was from the City of St. Louis. Not within the city limits but from the government of the City of St. Louis.

Response Note 1 – As of today, ITSA has not received any log information from Mr. Patterson or his website host. ITSA network engineers have requested these logs. Those logs certainly could provide our engineers more information on this activity.

SLP – I talked to one person later that week (I was the party initiating the conversation). We spoke briefly earlier in the afternoon and he said he was aware of the issue (I had privately talked to a few people in City Govt.). When he returned my call a few hours later he was dismissive, suggesting they’d have no way of tracking down such an event due to the large volume of traffic. At no point did he offer further assistance nor did he request the detail log.

Response Note 2 – The chart offered shows a volume of ‘5.51GB’ bandwidth utilized from a City of St. Louis IP address, with no time duration. From the chart, one cannot tell if it was over three minutes or three days or three months. ITSA’s total contracted bandwidth from AT&T is under 300MB, 1/18th the size stated on the chart. The City could not throw over 5GB of bandwidth at any server / website anywhere even if ITSA wanted to.

SLP – The amount of time was mentioned in my very next sentence!

Response Note 3 – The chart indicates that the event ended at 3:24PM on Monday, May 15, 2006. In other words, this happened in the middle of a typical business day. No City ITSA customers reported any internal network traffic problems at this time.

Response Note 4 – If all this dedicated bandwidth from a City IP address had occurred from within the ITSA managed WAN, all other services to internal City WAN customers would have failed and such events would have been logged. No such events were reported by any ITSA customers. No logs of service interruption at the indicated time have been recorded.

SLP – I think we need to compare logs. I want to see what their data shows for the same time period.

For about an hour and a half a server(s) asked for my main page at a rate of twenty times per second. At the time I characterized it as a deliberate denial of service attack.

Response Note 5 – Later in this posting, Mr. Patterson does state that fifty other websites are hosted upon this ‘attacked’ server. Without ITSA seeing any network logs or sniffer data, claiming that his site alone was the target of a planned DoS attack is not supported by any of the presented evidence.

I know a bit more now so let me share what I’ve been told. First, depending upon who you talk to you get a different answer — typical with technology issues. The chart at the right shows information on visits to my site all in mostly cryptic IP address. The top one, however, has been confirmed as being from the City of St. Louis. That IP is their standard outgoing IP for 42 various locations. As you can see the numbers are totally off the chart compared to typical traffic coming from many different ISP connections.

Response Note 6 – There is only one way in and out of the ITSA managed City WAN for public internet access by any ‘internal to the City’ ITSA WAN customer. The IP address listed is the blanket ‘public name’ of any ITSA City customer to the outside public internet. This is a common network management practice.

This is the IP address to the outside world presented by ITSA. Spoofing could be one possible explanation for the events described, since ITSA is physically not capable of generating the volume of traffic under discussion.

The city’s private security consultant did not want to characterize this as an attack. In fact, he said they can’t really track anything down because they have so many sites all using the same IP. I’ve been told attackers can sometimes “spoof” where they are coming from by giving a false IP address but apparently the type of tracking report my hosting company uses sees the real IP.

This leaves three scenarios.

Response Note 7 -actually, there are quite a bit more than just three scenarios – including some third party spoofed a City IP address; or that some other web page on the server was the target. What the City firewalls do track, by design, are failed communications attempts, and filtered internet traffic. This is common network engineering practice.

Some have suggested the city’s server just randomly messed up and began hitting a site by mistake, my site. Can you imagine the odds of that?

Response Note 8 – It is clear that the City WAN does not have sufficient data bandwidth to generate a DoS attack on the scale described by Mr. Patterson. If this attack did occur, it could not have come from the City WAN.

Another is that someone from outside the city’s network hacked into their system so they could launch the attack on my site and do it through the city’s system. That would be a scary thought that someone could do such a thing but I’ve been told it is not out of the realm of possibilities.

Response Note 9 – No one ‘hacked into their system’. There is absolutely no evidence that ITSA City WAN resources were compromised. As stated earlier, there is only one public way in and out of the ITSA managed City WAN, and that is fully monitored by two redundant firewalls.

The other, more realistic, conclusion is that someone did make a malicious attempt to knock out my site from within the system of the City of St. Louis. As I stated above, I’m told they have over 42 locations using the same IP address from the firewall.

Response Note 10 – There is absolutely no evidence, log or reported data communications within the City WAN at this time that indicates any support for such a statement. The nature of the IP address as presented to the ‘outside public internet’ was explained previously. Spoofed IP address DoS are unfortunately common occurrences.

Response Note 11 – The City’s network engineers, United Forensics, contacted Mr. Patterson on May 18, offering help, all of our data on the ‘event’, as well as an offer of 24/7 cell phone contact with our team if any future such event should be seen. No mention of this activity or offer of help by the City’s network engineers is mentioned by Mr. Patterson.

SLP – This is just plain BS. I talked with Josh Restivo briefly twice around May 18th and at no point were such offers made. I didn’t mention this in my post because frankly our conversation was a non-event. I felt dismissed. I think had they known the Suburban Journal and Post-Dispatch would cover the issue at a later date they might have been a bit more responsive to me. To date I have received no direct written communication from the city on this matter.

I’ve reported the abuse to SBC (AT&T), the city’s internet provider. I’ve gotten a response only to say they are looking into the issue. I’m not hopeful they will be anymore forthcoming with information than the city’s security consultant was.

Response Note 12 – Not true. United Forensics and the ITSA team has been very ‘above board’ in relating what we know, what we saw, how our network is engineered and managed, how much data bandwidth ITSA WAN capacity has and how it is allocated, and offered 24/7 help. ITSA and United Forensics offered to review the activity logs from his web host to aid in analyzing the event. The City takes this type of activity very seriously, and we stand ready to help in any reasonable fashion, and to review any and all log data.

SLP – Our two phone conversations — my initial call and the return phone call later that day may have totaled 5 minutes. They blew me off in May and ignored my post on the subject from June 2, 2006 (their responses here are to the June 2 post). It took a Post-Dispatch story on June 8 to actually get something of substance. Granted, I did not pursue them for any greater detail.

My site was slowed to the point of nearly being shut down. Sadly, the attack affected about 50 other sites on the same server including all the other blogs on the STL Syndicate and the Arch City Chronicle . The extra 5gb of bandwidth used by this attack does not come free.

Response Note 13 – As shown to date, a web service provider web site server was attacked, which houses by his own admittance fifty other sites. Without supporting log evidence, stating that ‘his web site was the target of a DoS attack’ is a jump to conclusion. The City and ITSA thank Mr. Patterson for bringing this type of activity to light, so that analysis can be performed and the City network security can be reviewed from a different angle.

Someone probably got a pretty good laugh over the whole deal but it shows a level of immaturity and fear that is unacceptable. If you don’t like my views write a well-reasoned opposing view but don’t resort to criminal activity just because you don’t like the message.

Response Note 14 – The total dedicated bandwidth for internal City WAN users to the outside public internet is 16Mb per second, far less than the 5.5GB presented as on the chart. ITSA has more than enough to do in addressing our internal City customer’s needs than to harass one blogsite. Any pre-disposed dedication of any internal ITSA bandwidth of this purported size in the middle of any business day to any an outside public internet address would be reported as service degradation by our customer community.

If that isn’t enough it seems the Mayor’s office felt the need to clarify the issue with the St. Louis Board of Alderman. Mayor Slay’s Chief of Staff, Jeff Rainford, sent out the following memo that same day:

To: St. Louis Board of Aldermen

From: Jeff Rainford

CC; Jim Sondermann, Ken Franklin

Date: June 8, 2006

Re: Post Dispatch Article


The St. Louis Post Dispatch this morning alleged that someone attacked a blog run by Steve Patterson from a City Hall computer. The Post-Dispatch reporter asked Alderman Jennifer Florida whether she was responsible for the attack. However, the Post-Dispatch did not ask us whether such an attack could have come from a City Computer. Had they asked, they would have learned that it did not, nor could it have come from a computer on the City network.

When I first heard about this, I asked Mike Wise, our director of technology, to investigate. If someone had done something wrong, we would have acted quickly and decisively.

Mike determined it was not logistically nor technologically possible for such an attack to have come from a computer on the City network. I have attached a copy of his response to Mr. Patterson’s allegations for your information.

I want to apologize to Alderman Florida. In my wildest imagination, I did not think the City’s only daily newspaper would make such an outrageous allegation without checking it out. If I had, I would have shared this information with you earlier. Obviously, I was wrong.

If you have any questions about this matter, you may feel free to contact me or Mike Wise.

Jeff Rainford
Chief of Staff.

You can click here to view a copy of the actual memo. I guess on the off chance someone at the Board of Aldermen didn’t know my name they certainly do now, thanks Jeff! But the part I’m stuck on is “outrageous allegation.” Is it really so “outrageous” to think someone within a major U.S. city government would be capable of such an attack? That the city’s network of hundreds, maybe thousands, of computers could accomplish such a feat? It cannot be disputed that my site was attacked and the evidence I posses suggests the city is to blame.

The P-D ran a story the next day, on June 9, 2006, to offer the city’s side on the attack issue. From the article:

The question, though, is whether the IP address was genuine, or a “spoof,” designed to make it look like the attack was coming from within the city.

“If somebody inside my network was responsible, we are going to find out who it was and act accordingly,” said Mike Wise, director of the city’s Information Technology Services Agency.

Wise said he doubts the attack, if that’s what it was, came from a city computer. The amount of bandwidth required for such an offensive would have slowed Internet access all over city government, he said.

“My phone would have been ringing off the hook,” Wise said.

Brian Marston, who provides Web hosting and support for Patterson’s site, disagrees. He says the city does have enough Internet power to enable an attack. He added that spoofing the city’s Web address would be unlikely – those type of maneuvers are typically reserved for major hack jobs.

I’ll let the computer folks among my readership debate the city’s claim of insufficient bandwidth as it is beyond my understanding. Maybe someone out there with more bandwidth than the city managed to attack my site and spoof their location to incriminate the city?

In the meantime I’m going to sit back and continue watching the various political maneuvering as officials come to grips with the fact they no longer control the local media. It is 2006 and the rules of the game are continually in flux as technology advances. Perhaps this whole event will serve as a wake up call to the suits

– Steve


Currently there are "11 comments" on this Article:

  1. Action Baby says:

    That’s a lot of information to digest. I am dissapointed though. I think you should pursue legal action, if only to get to the bottom of the situation and shed light on city hall’s possible criminal activity.

    If this is a common city practice (disrupt opposing views, intimidation) then you need to stop them! If they can do it to YOU then what will stop them from doing it to others. (didn’t city hall intimidate the save the century activists?)

    If it is highly likely the attack came from City Hall, I don’t think you should “…sit back and continue watching the various political maneuvering…”

    You’re doing a great job all in all, good luck and hope you run for alderman!


  2. Chris says:

    Ok, suppose the City’s services are incapable of delivering such an attack. Why isn’t the Mayor’s office more aggresively pursuing who ever is making them a patsy?

  3. travis reems says:


    As you know by our conversation earlier today, my following comments are in no way taking sides.

    That being said, with all the information I’ve read, if what the city’s network engineers say is true about their bandwidth, then I doubt it could have come from inside their network. If its not true, then it still would have taken multiple computers on their network to generate that many requests. Just as it slows the receiving server/computer of the attack, it slows, although not equally, the originating computer. So, this would have had to been a deliberate attack using multiple computers, unless the city got a Cray lately to calculate property taxes.
    More realistically, it was an outside attack using spoofed IPs, which is NOT reserved for big hack jobs. Highschoolers can do it.

    What is really surprising though, is the relatively amaturish way you were attacked–by simply requesting web pages. That’s not such a typical attack, like a SYN attack or ICMP flood. So, this is still all very odd.

  4. Jim Zavist says:

    TMI – he said, she said – yes, you got hacked, but we don’t need all the gory details and documentation! We feel your pain and don’t want to see this forum shut down! But, jeez, this is the longest post I’ve seen here and it involves cyberspace, not the urban environment!

    I agree that stifling discussion and trying to block opposing views is wrong, and I’m not surprised that there’s a bunch of bureaucratic CYA going on with your accusations/facts, but this appears to be a one-time attack, and unless it happens again, why devote so much space to an issue that’s off-topic on urban design issues? (Yeah, I know this is “your” site and you’re worked up, but try and keep things in perspective . . . )

    [REPLY – I agree it is off topic and way too much technical detail. But, you are free to skip over this post. Many other folks have been asking me privately about the details so I offer this for them.

    Also, while not directly urban environment related it does related to how politics & media work in this town. Other, like myself, find that interesting.

    But, I hear you. I will watch the amount of space devoted to posts like these. Thanks for sticking around! – SLP]

  5. action baby says:

    I agree with Jim to some extent that it’s just a lot of wasted space to devote to non-urban issues. Yet, in many ways it’s necessary.

    SLP, you are just preaching to the choir. I bet most of us here understand that your site was attacked. You should go after city hall and get to the bottom of this! What if you attacked city hall’s network?-would they sit back? No. NO. NO – the cops would be on your ass.

    That’s it

    [REPLY – Well, just like this post may be a lot of wasted space (can you waste internet space?) the time, money and effort required to pursue this issue further would only serve as a distraction to my focus on urban environment issues. I’m not dropping this topic but I’m also not devoting any financial resources to subpoena documents. – SLP]

  6. On May 15 between approximately 1 p.m. and 2:30 p.m. CST, 207-193-167-18.stlouiscity.com hammered urbanreviewstl.com, sucking up 109,324 pages, 109,853 hits and 5.51 GB of bandwidth. It was making 20 requests a second.

    My understanding is that the city has a 16 Mbps (megabits per second) connection to the outside public Internet.

    16 Mbps = 2 MBps (megabytes per second) = 120 MB per minute = 10.5 GB in 90 minutes

    So, 5.5 GB would be only about half of the city’s available bandwidth over a 90-minute time period.

    It’s technically possible that the requesting IP address was spoofed. Reasons I don’t think that’s what happened:

    1. My understanding is that spoofed denial of service attacks are usually directed at much higher profile sites (e.g. microsoft. com).

    2. The requesting IP address wasn’t just a random address. This was not a case of some hacker with no connection to St. Louis randomly attacking a site and making it look like the attack came from another random site. Either the traffic really came from the city, or somebody intentionally framed the city in an effort to make a statement about city government. The problem with the latter explanation is that they couldn’t have known that I would actually bother to look through the log files and figure out that is a city address, or that Steve would bring attention to it by blogging about the incident.

    3. If someone went through the trouble of spoofing an IP address to maliciously attack Steve’s site, why would he stop after one 90-minute burst of traffic only to disappear and not be heard from again? Why wouldn’t he continue to hammer the server until it was forced offline for a significant amount of time? As it was, Steve’s site was unreachable for only a few minutes. If you’re spoofing your IP address, the idea is that you’re making it look like someone else is to blame for the traffic and it’s hard to trace the attack back to you. I think that if it were a real spoofed denial of service attack, the attacker wouldn’t be hesitant at all about coming back day after day.

    4. I’ve been hosting websites for other people since October 2002, and this has never happened before to any of the other sites I host. If there were an easy way to get spoofed requests past my firewall, I would expect that this sort of thing would happen all the time.

    I sent the relevant portion of the Apache access log for urbanreviewstl.com to Michael Wise at his request on June 12, along with the explanation above. I haven’t heard back from him. I wrote him on June 15 to ask if he turned up anything, and he didn’t reply.

  7. For what it’s worth, Jeff Rainford was pushing Richard Callow to blog about the city’s response to this on MayorSlay.com. Apparently, he’s bent out of shape about it. I think Steve’s right that Jeff’s job was a lot easier before blogs when there were only a few easily cowed media outlets he had to smack down.

  8. IP Guru says:

    Honestly Steve, after reading this twice, just to be sure I “got it”, it looks like you are out in left field here. Really.

    The city clearly doesn’t have even a fraction of the bandwidth needed to do what you saw. I based this on both the city’s description (an admittedly strange one), and actual measurement tools on the internet: they appear to have a 14 to 18 mbps connection. While this is really fast for most of us, it’s not even close to what they would have needed to pull of this “attack”.

    One last thing as well. Like you, I’ve seen the letter you’re providing here – and whether you know it or not, it has been very much changed since it was delivered to you. I don’t know if it’s the match problem screwing up your reporting here.

    The city simply could NOT have performed the DoS described. Imposiible;

    [REPLY – OK, whomever mailed me the letter please send me any revisions so that I can see all the variations.

    As for the ability, see the comment two before with some match showing how it is possible. – SLP]

  9. This is possible. To trust the governments position so easily is quite moronic, especially with computers are considered, and lack of evidence is present. The City has not disproven that it is responsible. Where are the logs, and where is SBC’s report?

    16 megabits is quite fast, and as pointed out above, is significantly fast enough to transfer 10 Gigabytes.

    What should be remembered is that the 16-megabit connection is theoretical maximum throughput, and with collusions, and other outgoing connections, the true throughput would be slower.

    Bottom line: this is an attack; City Hall could have carried it out, yet there is not enough proof thus far to prove, or disprove the offender.

  10. Flaffer says:

    Doug Duckworth said:

    “Bottom line: this is an attack; City Hall could have carried it out, yet there is not enough proof thus far to prove, or disprove the offender.”

    Remember, there are access logs for the web server(s) that show all the hits from the city sourced IP. I would not call that “not enough proof”. I think the burden in on the city to prove that it was NOT their network that hosted the attack and they have failed miserably.

    I think there is plenty of prima facie evidence that the city is responsible, whether by accident or malice. They have not given any counter evidence except trust us and we cannot do the math (see Brian Marston’s post above.

  11. ed hardy clothing says:

    We'r ed hardy outlet one of the most profession
    of the coolest and latest ed hardy apparel, such as
    ed hardy tee ,ed hardy bags,
    ed hardy bathing suits, ed hardy shoes,
    ed hardy board shorts , don ed hardyt,ed hardy tank tops, ed hardy for women,
    ed hardy swimwearand more,
    ed hardy clothing. We offers a wide selection of fashion
    cheap ed hardyproducts. Welcome to our shop or just enjoy browsing through our stunning collection available wholesale ed hardy in our shop.

    our goal is to delight you with our distinctive collection of mindful ed hardy products while providing value and excellent service. Our goal is 100% customer satisfaction and we offer only 100% satisfacted service and ed hardy products. Please feel free to contact us at any time; we are committed to your 100% customer satisfaction. If you're looking for the best service and best selection, stay right where you are and continue shopping at here is your best online choice for the reasonable prices. So why not buy your ed hardy now, I am sure they we won’t let you down.


Comment on this Article: