Popmonkey.NET status

current status: normal

2012-01-06T11:30:00.000-08:00

POP3: normal
SMTP: normal
WWW: normal
DNS: normal
FTP: normal

mail server outage

2012-01-06T11:28:00.000-08:00

the mail server was down for 24hrs and 1 minute (:D) following a hardware failure. this has now been corrected. the down period was from 2012.01.05 09:18 PST to 2012.01.05 09:19 PST

because most mail servers will attempt delivery for 3 days no mail should be lost.

main host crash running again

2011-12-11T10:13:00.000-08:00

after 1 year and 10 days, the master host crashed this morning at 03:53 PST. recovery was done remotely at 08:31 PST.

this effected most of the public www services as well as the ftp server.

system is back online now and running diagnostics.

Update at 2011.12.11 10:13 PST: The server has crashed again so I need to go to the center to review what's going on. That will not happen until Monday.

Update at 2011.12.13 10:01 PST: New memory was installed and the server is running again

it's quiet... too quiet...

2011-10-03T15:52:00.000-07:00

HE.net has been fantastically reliable since we moved however they are getting *nailed* with a massive DDoS today. From their latest tweet:

henet: Experiencing large DDoS attack on our network. Network admins doing everything possible. No formal ETA at the moment. Will give update ASAP.

Update @ 16:30 PDT: HE says attack has been contained.

moved from godaddy to 1and1

2010-10-12T09:35:00.000-07:00

i moved domain hosting (for the domains i admin) from godaddy to 1and1. simply put, i had enough of goddady's chauvinistic bullshit. 1and1 seems just fine for popmonkey domain needs.

server move complete

2010-04-25T18:43:00.000-07:00

all the servers have moved and are operational. we had a few hours of DNS troubles due to some misconfiguration, network, and a chicken and egg issue...

servers moving

2010-04-25T10:34:00.000-07:00

servers will be down for a few hours today as we move them to a different data center. mail will spool up and be delivered when machines come back online.

Mail outage

2010-04-17T23:02:00.001-07:00

We had a 48 hour mail outage due to a problem with the virus scanner.

The virus scanner has been temporarily disabled until Debian Lenny updates clamav to 0.95.

Mail is flowing again.

quick power outage

2010-02-05T15:26:00.000-08:00

we had to power everything down in order to switch to a new power strip. everything will be back shortly.

update @ 15:31 : everything has been powered back up

ftp working again

2010-01-11T22:48:00.000-08:00

ftp is working again. if you didn't get your ftp instructions, email me

DNS problems this morning

2010-01-08T07:55:00.000-08:00

we had a DNS issue this morning with the primary DNS server. the secondary slave expired all the zones so there was effectively no DNS for a few hours. this has been corrected. sorry about that!

www/ftp getting fixed

2010-01-07T15:18:00.000-08:00

new server in place, www services will come back first (some domains are up already), ftp will follow next week.

i will email all the site owners when their site goes up.

www outage

2009-12-31T09:21:00.000-08:00

the web server responsible for most of the websites is down. no ETA on getting it going again because i totally not in the mood to spend nye working in the colo. why the hell do these things always fail around this time :(

services back up

2009-10-13T19:45:00.000-07:00

the power outage did some serious damage to the rack, but popmonkey stuff is ok now.

power outage

2009-10-13T13:34:00.000-07:00

servers have been down all morning - there was a power outage in the colo and when power came back on many servers did not restart. system are going up now.

www/ftp is back

2009-06-25T17:23:00.001-07:00

all systems normal again

problems with www/ftp server

2009-06-25T15:47:00.001-07:00

it seems to have gone offline - going to attempt to revive remotely. if that fails, i will not be able to get to it physically until tomorrow

connectivity has been restored

2009-04-09T14:55:00.000-07:00

what a mess!

we've lost all connectivity in our data center

2009-04-09T10:30:00.000-07:00

a fiber cable has been cut in the valley causing massive outages. our entire data center is now offline. no ETA on return.

heat wave = server shutdown

2008-06-21T12:00:00.000-07:00

another heat wave, another shutdown on one of the servers (the one that serves some of the websites and the nntp server).

the server is back now and i have once again requested an explanation from the datacenter. some of my drives were running 6 degrees Celsius over their safe temperature. not good although they seemed to have recovered. still, we might see more outages from this server.

Here comes the heat

2008-05-15T09:25:00.000-07:00

There's another heatwave in San Francisco and unfortunately the data center has heated up which is causing one of the servers to crash again as it did in mid April. This server really is just a finicky one - it's not nearly as hot in the cab as it was last time. I might put the server into some new hardware in the near future if things do not improve. In the meantime, expect some outages here and there. I will monitor and restart as needed.

serious problems with www/ftp server

2008-04-12T09:29:00.001-07:00

the server began crashing yesterday (2008.04.11) afternoon but was automatically recovered throughout the day. last night the machine crashed and would not restart. it looks very much like a trip to the colo is in order to fix it.

as this is a very old box running with very old disks in it, i am considering retiring it completely. the websites will all move to a new server but the news server will have to be retired as i do not have the bits to run it on other hardware.

more on that as things progress. i wont really know anything until i'm able to look at the machine in person.

update @ 13:22PDT: i'm in the colo now and the machine seems to have recovered after a lengthy disk repair task. it's extremely hot in here and i'm concerned that it's the heat in the cabinet that is causing problems with the hardware. the colo operators are working on bringing up another cooling unit as the place is definitely too warm for these machines. i'm leaving the server alone for now. if it crashes again i will take the disks out and bring them home and move all the data to a new machine. stay tuned.

update @ 14:33PDT: well that was wishful thinking. as soon as i got home the server went and died again. i will attempt a remote restart but i will not going back to the colo until tomorrow and then i'm not sure how long it will be before the www stuff is up again. note that the major www sites such as rideontwo.com are not at all effected by this outage.

updated @ 09:57PDT on Monday the 14th: i think the colo facilities temperature has dropped significantly and this server has been able to recover for the time being which points to an almost certain heat related issue with the hardware. I'm still planning on moving the data to a new machine but maybe it doesn't need to be so urgent.

mail server issues under control now

2008-01-27T10:31:00.000-08:00

I've installed some new hardware in the ailing server and we've been running very stable for the last week. I'm declaring victory for the time being

mail server issues continue

2008-01-19T19:14:00.000-08:00

in addition to the unstable server hardware, which actually has been hanging on pretty well the last few weeks, now the database server on the machine is crashing at various times.

this is almost certainly a hardware issue, the problem is figuring out which hardware component is causing all the problems. i'm not yet ready to replace the whole system...

mail server issues again

2007-12-29T12:16:00.000-08:00

The mail server is hanging again. I ordered some new memory for it which will hopefully fix the issue (definitely feels like bad memory).