Pretzel Logic Status

All Done

2007-04-15T19:27:00.000-04:00

The last of the maintenance was completed early this morning, so that should be the end of the small service outages.

Keep an eye on the main blog over the next day or two to find out the reason behind these upgrades.

Cheers.

Case Senitivity

2007-04-11T14:25:00.000-04:00

While we continue to make some updates for some up-coming features, it was noticed that
the mail server was rejecting some messages as "User Unknown" due to it being case sensitive. (so address such as JOHN.ROSS@pretzel-logic.ca would fail, while john.ross@pretzel-logic.ca would succeed)

Looking back at the logs over the past couple of weeks, very few messages were rejected, but since it is not the desired behavior, it will be changed later tonight.

Maitanance Over the Next Few Nights

2007-04-04T12:42:00.000-04:00

Over the next few nights we will be performing maintenance on the SMTP and IMAP/POP3 servers. This is to get it ready for some new features we'll soon be rolling out. (We're keeping the new features a secret for now, but announcements will be made over on the main blog when they're rolled out). During the maintenance, the SMTP and IMAP/POP3 servers will be intermittently unavailable. You may also occasionally get "incorrect password errors".

During this change we don't expect any mail to be lost, but it might be delayed. The work will be done between 9:00PM, and 1:00AM over the next few nights. I'll make an announcement here when all the work has been completed.

Done

2007-03-22T23:03:00.000-04:00

We started a bit late, but finished faster then expected.

All services are backup, and maintenance is complete.

Please let me know if there are any problems.

Cheers.

Brief IMAP and POP3 Interruption Tonight

2007-03-22T15:06:00.000-04:00

There will be a short interruption of IMAP and POP3 service tonight as we turn on indexing across all mailboxes.

This will allow searches in IMAP to be much faster. (If you're using POP3, you will see no difference as mail is not stored on the server)

Maintenance will begin at approx 10:00PM, and is expected to last less then 15 minutes. A note will be posted when maintenance is complete.

Cheers.

Services Restored

2007-03-13T23:04:00.000-04:00

All services have been restored as of 11:03PM

Service was down for just over 45 minutes.

We apologize of any inconvenience. Please let us know immediately if there are any problems.

E-mail service down

2007-03-13T22:34:00.000-04:00

E-mail and groupware services are currently unavailable.

A technician has been dispatched, and we expect services to be restored in about 30 minutes.

I'll post more information when service has been restored.

5 Minute Outage

2007-03-05T17:08:00.000-05:00

Due to some maintenance by our network provider, mail and one of our DNS servers were taken off-line for about 5 minutes today just before 5:00.

Everything should be back to normal now. We apologize for this inconvenience.

Cheers

Website will be down Thursday Dec 7th

2006-12-06T12:27:00.000-05:00

This website will be down briefly tomorrow morning for about 30 minutes starting at 10:00AM for some upgrades (more RAM)

This will not affect your service as e-mail and the groupware interface is on a separate server, but this blog, the RSS feeds and the support site will be down.

I apologize for any inconvenience.

Authenticated SMTP Now Available

2006-11-19T18:07:00.000-05:00

Authenticated SMTP is now available!

Authenticated SMTP allows you to use Pretzel Logic's servers to send mail from anywhere. Normally we restrict who can use our SMTP server to connections from our own network, and to other servers sending mail to our users. This is so spammers can not use our server as an open relay. This has the unfortunate effect of also keeping legitimate users who are traveling from using the server. Authenticated SMTP allows you to log in so that you can prove that you're authorized to send mail.

Also - many ISPs are blocking port 25 (The normal SMTP port) from your machine to everywhere except to their servers (Another anti-spam measure). This means even if Pretzel Logic's mail servers allowed everyone to send mail - most ISPs would block you. Fortunately, authenticated SMTP runs on a different port. This means as you move around from home to the office to your favorite coffee bar, you never have to change the settings on your mail client.

Here are the settings:

The outgoing mail server is outbox.pretzel-logic.ca
Your username and password is the same as for the IMAP/POP3 server
The port is 465

Your e-mail application likely has a section labeled "My outgoing mail server requires authentication", then you should see fields for the settings above.

As always, if you have any problems or questions, give us a shout and we'll be happy to help.

Cheers

IMAP authentication was broken

2006-11-09T11:03:00.000-05:00

Last night, one of our database maintenance scripts encountered a bug and prevented some domains from authenticate via IMAP.

The problem existed from about 6:15pm to 10:40pm.

We'll be working on the script today to permanently fix the bug.

During this problem all mail was delivered, and authentication to the web service continued to work. Only authentication to IMAP was affected.

Two Problems Found

2006-10-24T13:38:00.000-04:00

Good thing we kept an eye on things, we found two problems with the work that was done yesterday.

First, and most importantly - incoming mail was down until 13:27 today. The TCP ports were responding, so it looked like it working but mail was not being delivered. The problem was fixed, and should never happen again.

Second, when the RAID was reconfigured, we didn't catch that the drive assignments needed to change. Because of this we are only writing (and reading) to a single drive - instead of the RAID array. Unfortunately this will require another 90 minutes of downtime. Since this is a brand new drive, I'm not too worried about it, but it is something that will need to be fixed eventually in order to have fully redundant disks.

I'm really sorry about these problems. Our policy is however to be completely up front when these issues occur. They affect you and as much as I'd like to hide what is happening, or sugar coat the problems, you deserve to know what is happening to your service.

I also encourage you to comment, or email us if you have any questions.

Cheers.

Everything Checks Out.

2006-10-23T18:31:00.000-04:00

All checks are completed, and everything checks out - though we will keep an extra eye on everything over the next few days to make sure everything is happy.

I expect you might notice a speed increase as well. I suspect the bad drive was causing occasional slowdowns while the file system tried to deal with bad data. Of course the extra memory should help as well.

Let me know if you have any questions or notice any problems.

Cheers.

All services back up

2006-10-23T16:48:00.000-04:00

All services should now be back up, we're still checking, but so far,
everything looks good.

--
John Ross
Senior Systems Administrator, Pretzel Logic
John.Ross@Pretzel-Logic.ca http://www.pretzel-logic.ca

Services Will Be Down - Mon Oct 23

2006-10-22T18:49:00.000-04:00

In order to fix the drive problems we experienced a little while ago, we will be shutting down the system to install the new drives. We'll also be taking this opportunity to put in more memory (You can never have enough).

The machine will be shutdown shortly after 13:00, and we expect it will be back up by 17:00 (It will take a while to copy all the RAID data over to the new drives.)

The website, and this blog (http://www.pretzel-logic.ca/status) will stay up however, so you can check the status. (We'll update it when we're done)

Cheers.

Down the trail of bread crumbs

2006-10-11T17:02:00.000-04:00

Well, we've reached the end of the trail of breadcrumbs on Friday's outage.

Unfortunately they don't point conclusively at the cause of the problem, but there are suggestions it was a disk problem.

Because of this, in the next couple of weeks we'll be installing a new disk into the server. There will be a few hours of interruption while this occurs. We'll give you a further heads up with specific times when we get closer to the actual installation.

If you have any questions, please let us know.

Cheers

Investigation continuing

2006-10-06T14:06:00.000-04:00

All services were backup by 1:30PM. Investigation is continuing, but it looks like there was a disk issue. None of the disks are reporting any errors, so at this point it's a guess.

We'll post an update when we've completed the investigation.

Groupware Services are down

2006-10-06T11:23:00.000-04:00

At approximately 10:55AM this morning, the primary groupware server became unresponsive.
As this includes SSH, we're unable to determine what is going on. (After talking to our provider we've ruled out DoS attacks, or network problems)

We'll be visiting the site shortly, and will send updates as we know more.

Timeouts

2006-09-26T11:35:00.000-04:00

A few users reported timeouts this morning - around 9:45, and a couple of automated systems registered that something weird was going on.

From what I can tell, it seems there were some network issues this morning upstream of Pretzel Logic.

It seems that the problems have sorted themselves out, so you shouldn't get any more timeouts.

Cheers

Aliases

2006-09-08T11:00:00.000-04:00

Aliases are now working.
Aliases allow you to have multiple addresses delivered to a single mailbox. You may have up to three addresses per mailbox (Contact us if you need more)

To add the second and third address, select the "Contacts" application and select your entry.
Click on the "Edit" button on the top right, and then enter your other addresses in the appropriate fields. (Your primary e-mail address must be listed under "Email1")
If only Email1 and Email2 are listed, you can click on the "configure" button from the "Person Viewer" page, then select what additional fields you would like to see.

New addresses will be available within five minutes.

Cheers.

Server crashed

2006-08-15T11:43:00.000-04:00

One of the servers crashed this morning at about 02:17.

haven't found the actual cause yet, but we're tracking it down.

UPDATE:
The problem was caused by the storage subsystem - specifically the USB system.
To fix this problem we will be rebooting this server again at 11:30PM tomorrow (Aug 16). While the machine reboots (for about 5 minutes) the web interface, and mail will be unavailable.

UPDATE: Thu Aug 17 00:26
The machine was rebooted, and all looks well. The problem should be no-more.
All services are now accessible.

Demo Signup Exploited by Spammers

2006-08-09T13:27:00.000-04:00

Erp.

The demo signup form on the front page was exploited by spammers recently. It looks like only one message went through (likely a test message). The problem is fixed and should have no effect on normal use, but if you do see problems let us know.

Logout issues

2006-08-01T20:09:00.000-04:00

Some users are reporting errors after clicking on the "here" link on the logout page. We're looking into this, but if you experience this error, you can hit the reload button of your browser to return you to the login page.

Website upgrades

2006-07-26T17:28:00.000-04:00

Over the next few days, we'll be making some changes to the website. This may affect your groupware login/logout pages as well. These changes should not affect the actual interface.

The individual changes should be quick, but may keep the page from displaying properly if you happen to be loading the page just as the changes are being made. If you've been affected, just hit the reload button of your browser, and the page should come up.

Mail Authentication down

2006-07-20T15:46:00.000-04:00

Customers were unable to access their mail for about 30 minutes when we encountered a bug with the SQL scripts that keep the login information in sync between the servers.

This did not affect mail delivery - only the ability to pickup your mail.

A permanent fix for the bug is now in place, so we should not see it again.

Everything was back to normal at 3:37 PM