Edoceo's Blog

The Trouble with NetSol

2012-01-02T13:44:00.000-08:00

One of our clients hosts their domain and DNS with Network Solutions ("NetSol") (their first mistake).
For those who don't know: NetSol is the oldest, more arcane domain services provider.

Today, we wanted to update some DNS records (A, CNAME). An easy task with other providers such as eNom or GoDaddy.
Here is the story.
For references "@" is the domain name in question, a common

Cannot Delete A Record for www.
Our first step was to remove the A record for the www. entry, which we wanted to replace with a CNAME.
In the DNS manager provided by NetSol we check a box labelled "Delete" and press save.
The record was not deleted, the IP address portion was, but the entry it-self was not.
The result was that a lookup for "www.@" was simply empty. WTF?

We contacted support, and after more than 20 minutes of navigating the IVR and waiting we were connected with a CSR who mumbled and didn't know a freaking thing about DNS.

We explained to them our issue, which they didn't understand.
After repeating 'We want to delete the A record for WWW' about a dozen time they got it.

The CSR then looked at our account, made some adjustments and asked us to verify.
All the entries were gone! What the FUCK?!
Except, that the 'A' record section of their manage won't allow us to remove the entry for 'www', it's hard coded.
What an epic fail.

So, now we had to re-create all the A records the previously existed.
Good thing we had documentation.

So we tried the whole process again.
Checked the "Delete" box next to www again and selected delete.
Waited for their slow caching system to referesh the data (five minutes) and saw the same results.
The A record persists, with a blank entry, and prevents us from creating a CNAME record for www..

In short, the DNS "Manager" provided by NetSol is a flaming pile of crap.

How to Fix It

Firstly, NetSol, could improve their system so that it actually functions, uses caching properly and allows the user to manage A, CNAME and other records with full authority.

Another option is to choose an alternate DNS provider - such as that provided by eNom or GoDaddy (or dozens of others).

For us, we assisted the customer with migrating their domains out of NetSol.

Moving Mail via IMAP from Account to Another

2011-12-16T05:55:00.000-08:00

Most of the hosting providers these days provide for IMAP access to your mailbox.
This is great, but what if you want to move messages from one IMAP account to another?
We wrote this little tool for it: imap-move.

The IMAP Move tool was written with a few, very specific, intended purposes but there could be others.

Archive messages from IMAP into a local directory
Move message from one Mailbox to another Mail Account

The tool keeps track of the moved message (by mail message ID) so it can be run over and over on the same source mailbox w/o having to repeat it's work - this can save time when you have 40k+ messages.

List the Source Folders/Labels

imap-move.php -s imap-ssl://user@domain:password@imap.gmail.com:993/ --list

Source: user@domain
01 Follow up; 0 messages 0 bytes
02 INBOX; 1080 messages 108104263 bytes
03 Misc; 0 messages 0 bytes
04 Priority; 0 messages 0 bytes
05 [Gmail]; skip [container only]
06 [Gmail]/All Mail; skip [skip list]
07 [Gmail]/Drafts; skip [skip list]
08 [Gmail]/Sent Mail; skip [skip list]
09 [Gmail]/Spam; skip [skip list]
10 [Gmail]/Starred; skip [skip list]
11 [Gmail]/Trash; skip [skip list]
11 Folders, 1080 messages, 108104263 bytes

Copy to Local Directory

Please note, this is not a MAILDIR (or, at least I don't think it is)

imap-move.php \
  -s imap-ssl://user@domain:password@imap.gmail.com:993/ \
  --copy /path/to/storage

Copy one IMAP Account to Another

This can take a long time, the application has a lot of spew, maybe use |tee or something.

imap-move.php \
  -s imap-ssl://user@domain:password@imap.gmail.com:993/ \
  -t imap-ssl://noob@domain:password@imap.gmail.com:993/ \
  --move \
  --copy

A Week without Flash

2011-12-09T10:48:00.001-08:00

Adobe Flash is a flaming pile of crap.
Firstly: it's got loads of security issues and consumes hella memory (>=100M!).
Additionally: ads and tracking issues, impact on web-surfing performance, &c.

So starting Monday of this week we to a (seemingly) dramatic step - removed Flash from our systems.
Not use using FlashBlock (which is awesome!), not just disabling the Plug-In (also an option); fully gone.

Must say, I've hardly noticed the difference.

Many pages, especially on ad-ridden sites (looking at you Cheeseburger), load much faster.

Adobe Flash Fail Sites

Not everything was great. To watch YouTube we had to enable their HTML5 beta.
Also, some functionality of both Google Analytics and Google Finance depend on Adobe.
Lots of other stuff does too.

Important to Remove

If, like us, you want to see the death of flash!, start removing it.
This way, we can start to skew the analytics numbers to show less and less browser support.

Old, crappy technologies don't die out on their own, we have to actively kill them.

Getting Started with PhoneGap on Linux without Eclipse

2011-11-30T16:18:00.001-08:00

Eclipse is a great editor and all, but I'm a much bigger fan of jEdit which I've been using since forever (at least 10 years). So, when wanting to do Android development and especially when using PhoneGap most of the existing documentation focuses on Eclipse.

This shows a full manual process, from installing the Android SDK and PhoneGap, creating the initial project and building the sample provided by PhoneGap. The target system is Linux (Gentoo) and we'll mostly be working in the home directory of a non-privileged user.

Install the Android SDK

At this time of this writing it was version: android-sdk_r15-linux.tgz.

Might take some time to complete all the downloads during update sdk.

wget http://dl.google.com/android/android-sdk_r15-linux.tgz
tar -zxf android-sdk_r15-linux.tgz
rm android-sdk_r15-linux.tgz
cd android-sdk-linux
tools/android update sdk --no-ui
echo "PATH=\"\${PATH}:~/android-sdk-linux/tools\"" >> ~/.bashrc

Install PhoneGap

Download the latest, at this time PhoneGap 1.2.0.

wget https://github.com/callback/phonegap/zipball/1.2.0
unzip 
cd phonegap-1.2.0

Building the Sample

cd ~/phonegap-1.2.0/Android/Sample
android update project --path $(pwd)
ant debug

Running the Sample

~/android-sdk-linux/tools/android

Using this UI create a AVD Tools -> Manage AVDs

~/android-sdk-linux/platform-tools/adb devices

Now install to the running emulator

~/android-sdk-linux/platform-tools/adb -e install -r bin/PhoneGapExample-debug.apk

Credit Card Brand Logos as CSS Sprite

2011-11-06T15:13:00.000-08:00

We recently had cause to create some CC icons for a checkout process, we just needed the four most common. Had a heck of a time getting good downloads, quality images &c. Google has a good image they use; one image file used as sprites - smart.

Other sites the downloads were not working, or there was some restrictive license on the images. The card vendors themselves also don't make this easy - requiring registration to download? WTF? Why not post to their Facebook pages, make downloadable from the Pictures section.

So, we searched and went directly to each brand, downloaded their source images and created this image sprite, as a indexed- PNG so it's roughly 8K! It has a hot and cold view of each of the four major credit card providers. Use as needed, no-fee, no-charge, no-license, no-restrictions.

To use it, do something like this:

.cc { height: 36px; }
.cc div { background:url(http://cdn.edoceo.com/img/cc.png); border:1px solid #666; float:left; }
.cc .mc { background-position:0px 0px; height:32px; margin:2px; width:56px; }
.cc .vc { background-position:-56px 0px; height:32px; margin:2px; width:56px; }
.cc .dc { background-position:-112px 0px; height:32px; margin:2px; width:56px; }
.cc .ac { background-position:-168px 0px; height:32px; margin:2px; width:56px; }
.cc .mb { background-position:0px 32px; height:32px; margin:2px; width:56px; }
.cc .vb { background-position:-56px 32px; height:32px; margin:2px; width:56px; }
.cc .db { background-position:-112px 32px; height:32px; margin:2px; width:56px; }
.cc .ab { background-position:-168px 32px; height:32px; margin:2px; width:56px; }

<div class="cc">
  <div class="mc"></div>
  <div class="vc"></div>
  <div class="dc"></div>
  <div class="ac"></div>
</div>

<div class="cc">
  <div class="mb"></div>
  <div class="vb"></div>
  <div class="db"></div>
  <div class="ab"></div>
</div>

Add some JavaScript card type auto-detection and adjust the card indicators as necessary.

Thanks to merchant account blog for some pointers. More information about CSS Sprites can be found on CSS Tricks.

Network Solutions DNS Migration - Caution Parking!

2011-09-16T18:34:00.000-07:00

We recently assisted with a migration of DNS services to the Network Solutions (worldnic.com) which is a pretty normal operation.

The process went as normal, transferred the domain to Network Solutions ("NS"), waited a few days for that to process. Once in Network Solutions the WHOIS information was updated and again we waited 48 hours for the next move.

A zone-dump was acquired from the existing DNS provider, including all A, CNAME, SRV and TXT records. We initiated the DNS migration (Switch NS) process at NS and then proceeded to update their DNS with the proper records. After two hours the settings were verified via direct query using dig. All was well.

About eight hours later the client called, they had noticed an issue on the web-site. Imagine our surprise when, upon examining the DNS records we found two new records that we did not create!

Apparently, some time between 11pm and 7am these two records were added. The two A records were for @ (i.e.: domain.tld) and www. They pointed to a parking page IPv4 address run by NS. Our existing records were still in-place.

The client was seeing the issue due to the round-robin effect of these two addresses. Some times seeing the proper site/host and other times not.

Simply deleting these spurious records and waiting 7200 seconds did the trick.

McAfee "Secure" / HackerSafe - Bad Config?

2011-09-01T11:16:00.000-07:00

For some of our clients that require PCI compliance we had been using the "McAfee Secure" (aka: HackerSafe) product. Changes to their site over two years ago eliminated the need for this snake-oil type service - so we canceled the service circa 2010q1.

Oddly, their service has continued to scan and send "alerts" for our system.

We contacted McAfee regarding this issue (877-302-9965) and confirmed that our account had been closed "for some time"

Attempts to login to the McAfee control-panel failed, as did attempts to reset our password using the known/proper email address on file. We were told by the system: "A mail has been sent to the email address" - cute. We waited two hours for this message.

It appears, that like so many other snake-oil security products McAfee's system will also repeatedly nag you about spurious events.
So, just be aware and careful.

Internally it's a good idea to use tools like OpenVAS to check your systems as well as a third-party provider.

Stop Wasting Time with Social Media

2011-08-18T11:53:00.000-07:00

Social Media has been a hot marketing topic for some time but it has very little value! Don't sink more than five hours a week into this low-value advertising.

A study by Outbrain says that Social Media lags for Traffic Sources. The results of that report has a small delta to most sites we manage.

Search is far and away the leader (41%), second is content sites (think Digg, Reddit). Portal (AOL, MSN)and Social are third and fourth with a sum-metric of 28%.

Our opinion is that driving Social Media should be a third order priority. At this time simply posting new products, service announcements, company news & links to Social Media is sufficient. The traffic-source numbers are just not there. There are eye-balls yes; but they don't engage as much.

All this of course will change over time. Until Social Media has some real traction our advice is to cover the basics, get the branded accounts/pages and make some posts. Keep the primary focus on Search Optimization then Local (Places, Yelp).

Credit Card fraud and New Egg.com - Update

2011-07-01T11:59:00.000-07:00

Credit card fraud is a scary deal, could be a one-time deal - or could be the start of identity theft - which is really serious. Fortunately this was a one-off.

Spoke with NewEgg on the subject, as follows:

Circumstances were very suspicious, will take some internal action/review. Reviewed account and previous orders, to ensure that everything was proper. Put a lock limit on my orders (I'll have to double verify certain purchases). Split charges is due to fulfillment and tax requirements.

The CSA seemed to have authentic concern for the situation assured me they would look into it.

Single Use Credit Cards / Virtual Account Numbers

Learned about a number of new options for single-use credit cards - widely known as "Virtual Account Numbers".

Bank of America Shopsafe - http://www.bankofamerica.com/privacy/index.cfm?template=learn_about_shopsafe

Citi appears to offer this as VAN: https://www.citibank.com/us/cards/vanpromo/cmc_pop/index2.htm

Chase and Wells Fargo did not have similar offerings. Heard rumors that Paypal and Discover might offer something, if you know - post a link!

Credit Card fraud and New Egg.com

2011-06-30T17:18:00.000-07:00

We recently purchased roughly $750 worth of goods from NewEgg.com and two days later the card used for that transaction was being declined at every purchase!

After being embarrassed when a purchase of a $4 coffee was declined we went to the bank to investigate. We were then informed that there were some fraudulent looking activities against our account so the card had been frozen. Good thing.

What made the transactions look like fraud was initially that NewEgg hit the account two different times - once for about $400 and the other for close to $350 - with just a few short hours between the to transactions. These transactions originated from out-of-state - I'm in WA, NewEgg is in CA.

The final straw for the fraud detection was a transaction from a Wal-Mart Bowling Green, KY.

What the Fuck!

So, in the first place those two charges from NewEgg triggered the FDS - which is what allowed Wells Fargo to detect the bull-shit charge from Kentucky. Thank you WF.

But how could my card number have ended up Kentucky? Well, on speculation here's what we have.

NewEgg (who processed that transaction) has an order fulfillment center in Memphis, TN (as evidenced by the UPS tracking number). Those shipments travel on I-40 from Memphis to Nashville, TN then north on I-65 to Lousiville, KY - right, directly through Bowling Green, KY - where the fraud occurred.

So, some how, a purchase from NewEgg on the 28th, triggered a fradulent credit-card transaction on the 29th - while my package from the order was traveling through a city on it's route at that same time! What are the odds!?!

Worst case NewEgg is complicit in credit card fraud. Best case - they have some very unscrupulous people working there.

Also, congratulations to Wells Fargo for detecting and blocking this fraud before it became more than a few hours of inconvenience.

Now I'm stuck waiting for five to seven business days for replacement cards - I have to get cash from the bank teller - like a caveman!

Google Apps Transition: Checkout

2011-06-10T00:56:00.000-07:00

As a crappy alternative to being able to properly transition this seemingly simple set of data you can leave the @gtempaccount.com in place and assign to a secondary Admin in your Apps account - at least until Google fixes this fail

For now all we can do is Tell Google: "Allow me to Transition this data to my Google Apps based Account!@1!" Visit the suggestions page

Google Apps Transition - Google Analytics

2011-06-10T00:28:00.000-07:00

In similar fashion to the issues with the Blogger Service, Google Analytics require similar steps to remove the gtempaccount.com association - and re-join into the proper/original account in your Domain.

Sign-in to Analytics as the user% account
Select User Manager at the bottom of the dashboard
Select Add User in the upper-right of the user table
Enter the email address of user@, set Access Type to Account Administrator and enable all Website Profiles.
The new user won't get any email notification
Sign-out and flush cookies, Sign-in as user@ account
Use the User Manager to remove the user% gtempaccount.com user

A little easier than the Blogger process and all history and identifiers will be preserved.

Google Apps Accounts Migration for Blogger & Checkout

2011-06-09T20:50:00.001-07:00

Google has updated their Apps and is forcing all users to Migrate. Unfortunately not all of their services are ready for this and will not fully migrate automatically.

At the time of this writing the following services fail to transition properly: Analytics / Website Optimizer, Blogger, Google Alerts, Google Base, Google Calendar, Google Checkout, Google Custom Search, Google Docs, Google Maps, Google Moderator, Google News, Google Reader, Google Subscribed Links, Google Talk, Groups, iGoogle, Places and Webmaster Tools.

Yea, that's quite a lot, these 18 products (of roughly 70) are not ready. Google has mentioned that eventually this will be fixed but has not announced any ETA.

That makes it difficult for anyone, including us and our clients, to plan. We've been forced to figure out work-arounds and here is one for what to do on Blogger.

Migrating Blogger or Checkout with Google Apps

This is a tedious process, where your Blogger or Checkout account is going to have the existing authors switched around. You will need two Google Apps accounts, in the same domain, to perform these steps.

Is is critical that you do not delete your blog or you will lose the custom domain (like blog.edoceo.com)

user% represents the old account with the conflicting name that was migrated to gtempaccount.com
user@ represents the new account in the Google Apps domain
temp@ represents the account in the Google Apps domain which will be used for an interim Author

There were some transitioning issues when attempting to go from the user% account directly to the user@ account - so we had to use an interim transitioning account - which temp%

E.G. For us user@ would be our old blog@domain.com (user@) account, which got migrated to blog%domain.com@gtempaccount.com (user%). We had to use the interim account shit@domain.com (temp@) during this migration.

Here are the steps to follow, may the force be with you.

Sign into Blogger using the user% account
Visit Settings » Permissions and select to add a new author
Enter the account for temp@ to become an Author
Sign out of Google as user%, flush cookies and sign-in now as temp@
Follow the link in the invitaion email, connect to Blogger and create your Profile
Sign out of Google as temp@ account and sign-in again as user%
Grant Admin Privileges to this the temp@ user account
Sign out of Google as user% and then sign-in as temp@
Go to Blogger Settings » Permissions again and remove the Author user%
Sign out of all Google stuff and flush cookies
Wait about 20 to 30 minutes for the Google Infrastructure to update, going too fast can make the next step fail.
Did you really wait 30 minutes? Do yourself a favour and execute this step properly
Sign-in as temp@ then add user@ account to your Blogger (Settings » Permissions), this will bring in the Apps account.
Sign-out as temp@ then sign-in as user@ and follow the link in this email.
Sign-out again as temp@ and promote user@ to an administrative level account
Sign-in now as user@ and visit Blogger, you should have full control.
None of the Profile settings from the user% account will be migrated to the user@ account so you will be required to re-create all that.

There is some data migration issue that Google has not resolved yet. When you start promoting your Google Apps domain to function more like a Google Account (or when Google forces your migration) these steps will be necessary to clean data out of your @gtempaccount.com domain and into your desired domain.

Google Apps to Accounts Transition Failures

2011-05-05T12:01:00.000-07:00

The process of migrating Google Apps to their new Google Accounts features is an un-mitigated disaster. Our clients and ourselves have had nothing but problems the whole time. The transition process is incomplete, orphans many critical services and has no work around.

If you use any of the following services you'd want to avoid migration but Google is forcing us along that path - so now your vendor (Google) is shooting you in the foot.

Checkout Fails to Migrate

Google Checkout fails to migrate and as of yet there is no option to transfer data, so you are forced to create a new Google Checkout, re-validate your bank-account and re-configure your store front an everything, effectively starting from scratch. Major Fail.

Analytics, AdSense and AdWords Fail to Migrate

These services, also critical for those of us paying for Google Apps also fail to migrate leaving orphaned accounts. Start all over.

Blogger Fails to Migrate

Blogger fails to migrate but you can switch owners, a simple manual process. One would think that something so easy could have been built by Google's Engineers before they attempted to push us all to migrate.

They also provide a page with instructions, but those don't work yet. From our experience and those we've tried to help, you cannot add the transistion account because the target domain account is already listed as an Author, but when you login with that account you cannot make posts. Sloppy at best.

Alerts Fail to Migrate

This one also fails, forcing humans to manually re-create the alerts they had.

Google Apps, Talk / XMPP, Andriod Failures - Part 2

2011-05-05T11:54:00.001-07:00

The process has now gotten even worse. We have had to visit the Google Apps domain control panel multiple times now to disable Chat, and it some how keeps getting re-enabled. The "support" staff from Google is still clueless, after more than a dozen detailed emails showing direct XML between their XMPP system and ours.

Update from Google

We spoke to Google about this issue, apparently the Android team knows this is an issue but, well you can see:

Google: "They dont have an ETA yet, they told me they are still working on it and it will be sometime in the distant future. They cannot give an exact date or time period because there is a lot of testing and development to be done etc and all the various variables involved make it hard to give a time period. But they said they are working on it and will try to release some of the fixes in future updates to the OS. Hope that helps."

Update from Edoceo

The previous fix we listed actually does not work. Oddly what happened was that Chat/Talk got re-enabled for our domain when we told our Android phone to sync; that then caused problems for everyone else who was trying to chat with us. It's broken and Google doesn't care about their customers, no ETA on fix.

Google Apps, Talk / XMPP, Andriod Failures

2011-04-28T11:27:00.000-07:00

We recently decided to disable the Google Talk feature for our Google Apps for Business and use our own Jabber/XMPP server to improve monitoring and audit-ability of these communications. Turns out Google does not like it when you dis-connect their services and it will fully break email services on your phone.

Disabling Google Talk / XMPP

This process is simple enough.

Create your own XMPP Server (ejabberd works well)
Update the DNS Service Records
Disable the Google Talk service in the control panel for Google Apps

At this point you'll notice that your Android phone starts throwing authentication errors for Google Talk, even if the Talk service is not running, disabled and such. This error about Talk authentication is actually coming from the Google Mail service on the phone. It appears every time the phone tries to sync the mail or calendar.

We had hoped this issues would be resolved after all systems were updated but even after 24 hours these errors/issues were still affecting all users.

The actual error is this:

Authentication Error
Google Talk failed to login. If
this is a Google Apps account,
confirm that Chat service is
enabled for this account.
  [ Retry ] [ Cancel ]

If you press Retry it will of course retry and fail, if you press Cancel then your phone will no longer receive mail on that account, leaving you bricked. Don't worry, it will throw the error again in a few minutes - and still not deliver email to your phone. This error still appears even when Sync is fully disabled for the affected account.

Attempts a Resolution

We attempted multiple ways to try to fix this. The easiest fix was to simply remove that Google Apps account from the phone and re-add it - all was well.

However, that does not work on the primary Google Account on the phone as that one cannot be removed! Now you have a broken account that cannot be removed, re-created and updated to function properly.

So we opened a trouble ticket with Google, via email, and waited 36 hours without a response. As we have Google Apps for Business we also get telephone support for "down" emergencies, such as when our phones cannot receive any email messages.

Three times we called Google (877.355.5787) and were disconnected by their system after entering our support PIN, that sucked. We finally got a hold of them to explain the issue, went like this:

Me: My phone no longer receives messages. I disabled Talk services and now phones cannot receive email.
Google: I'll have to dispatch this to our technical team
Me: [ waits 10 minutes ]
Google: What network connection are you using wee-fee?
Me: You mean why-fi?
Google: Here is your ticket number, we'll call you back within 24 hours. Anything else we can help you with?
Me: It would help to get support now, not some mythical time in the future, but we have no choice, I guess I'll wait with broken email.

They sent an email to follow up with my email down issue, awesome. Of particular note was that our domain name was listed as "Edoceo, Inc.", affected accounts were mis-identified, and the carrier was noted as "Striant" - not "Sprint" like I explained and spelled for the CSR who took the call (their command of the English language was not good).

Hard to have confidence in Google when the service is so fragile and the "support" team is so very sub-par.

How to Fix (what Google should do)

Google Mail should not be attempting to sign-in to Chat when Chat is disabled for this domain. Perhaps it could update settings on the account when service options change.
An option to disable this would be handy. There are no settings for the Chat service in the Google Mail account setup on the phone, only for Contacts, Mail and Calendar.
Not forcing users to tie their phones to an un-removable Google Account would be handy.
Calling the product by a single name would help, Chat and Talk are the same, but confused the CSR.

Update from Google #1

Google advised to just add the account (yes, the one that was already added and that could not be removed because it was the primary). So their first attempt at "support" was #fail. Clearly their support reps did not even read the report we submitted. Also of note here is that follow-up to our support telephone call was an email from a support rep that only contained canned instructions. Google clearly has a lot to learn about customer service.

Update from Google #2

Supposedly can simply 'Resync Account' by visiting Settings » Accounts & Sync » [ your account ] » Sync Now. We tried that, waited but the error still appeared.

Update from Edoceo #1

That clue from Google may have started the right process. We noticed still at issue was the Google hosted servers still (even after 24h) think they are responsible for XMPP services for edoceo.com (details below). While that is not true, their updates appear to just be taking a while to publish to all their systems. A typical problem for large scale service providers.

Fixed: Clearly caching is an issue, which led us to the following solution.

Settings » Accounts & Sync », for each affected account
Uncheck All Sync Options (Contacts, Gmail, Calendar)
Wait for two hours (could be less, but that's how long we waited)
Revist account options, re-enable Sync for desired services

This will even work to fix issues on the primary account, which cannot be removed. Once it came back on-line the errors from Google Talk Authentication were no longer present.

On another device, which had the affected account not the primary, we were able to remove and re-add the account to fix. Duh, right?

Here is the details about the issue from our eJabberd verbose logs.

<stream:error>
  <undefined-condition xmlns="urn:ietf:params:xml:ns:xmpp-streams" />
    <str:text xmlns:str="urn:ietf:params:xml:ns:xmpp-streams">edoceo.com is a Google Apps Domain with Talk service enabled.</str:text>
  </stream:error>
</stream:stream>

<iq from="$buddy@gmail.com" to="me@edoceo.com/Office" type="error" id="purplef88a9f67">
  <vCard xmlns='vcard-temp'/>
  <error code='404' type='cancel'>
    <remote-server-not-found xmlns='urn:ietf:params:xml:ns:xmpp-stanzas' />
  </error>
</iq>

Our Best Uptime - Ever!

2011-03-24T04:03:00.000-07:00

At Edoceo we've been operating servers and internet systems since at least 1997. Many of us have experience dating even further. This month (Jan 2011) we'll be moving systems around and one of our longest running systems will have to be taken off-line forever - the hardware is simply too old now.

Just sharing a bit of our history on this box and a little bragging too.
This system has been running since Wed Feb 1 02:25:22 2006!!

The system itself is an old P3/850 that we purchased at a dot-com liquidation auction run by James G Murphy auction house - we attended at least a dozen of their auctions - they run good operation. It was installed with our favourite distro: Gentoo and housed at SiteSpecific in Seattle.

We've just finished migrating all services off and are about to issue the final `poweroff` command once this is posted.

The Hardware

It's a P3/850 with 512MiB of RAM and a small, 15GiB IDE disk.

lithium ~ # cat /proc/cpuinfo 
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model  : 8
model name : Pentium III (Coppermine)
stepping : 6
cpu MHz  : 847.583
cache size : 256 KB
fdiv_bug : no
hlt_bug  : no
f00f_bug : no
coma_bug : no
fpu  : yes
fpu_exception : yes
cpuid level : 2
wp  : yes
flags  : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1697.27

lithium ~ # cat /proc/meminfo 
MemTotal:       515424 kB
MemFree:        283108 kB
Buffers:        156472 kB
Cached:          36652 kB
SwapCached:          0 kB
Active:         179176 kB
Inactive:        18396 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       515424 kB
LowFree:        283108 kB
SwapTotal:      498004 kB
SwapFree:       497948 kB
Dirty:               0 kB
Writeback:           0 kB
Mapped:           9488 kB
Slab:            33132 kB
CommitLimit:    755716 kB
Committed_AS:    11856 kB
PageTables:        224 kB
VmallocTotal:   515816 kB
VmallocUsed:       952 kB
VmallocChunk:   514788 kB

lithium ~ # sfdisk -uM -l /dev/hda

Disk /dev/hda: 32760 cylinders, 16 heads, 63 sectors/track
Warning: The partition table looks like it was made
  for C/H/S=*/255/63 (instead of 32760/16/63).
For this listing I'll assume that geometry.
Units = mebibytes of 1048576 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start   End    MiB    #blocks   Id  System
/dev/hda1   *     0+    62-    63-     64228+  83  Linux
/dev/hda2        62+   549-   487-    498015   83  Linux
/dev/hda3       549+ 16119- 15571-  15944512+  83  Linux
/dev/hda4         0      -      0          0    0  Empty

Software

Very old 2.6.x kernel, reasonably new OpenSSH, Lighttpd and Postfix

lithium ~ # uname -a 
Linux lithium 2.6.13-gentoo-r5 #1 Tue Nov 1 14:42:07 PST 2005 i686 Pentium III (Coppermine) GenuineIntel GNU/Linux

lithium ~ # ssh -V
OpenSSH_5.2p1, OpenSSL 0.9.8k 25 Mar 2009

lithium ~ # lighttpd -V
lighttpd-1.4.20 (ssl) - a light and fast webserver
Build-Date: Jun 23 2009 07:51:59

lithium ~ # postconf mail_version
mail_version = 2.5.5

The Uptime Proof

Here shows our uptime command, process table, tune2fs information.
Notice the kernel started in 2006!

lithium ~ # uptime
 11:38:42 up 1804 days, 16:42,  2 users,  load average: 1.05, 0.73, 0.45

lithium ~ # ps -eF
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD
root         1     0  0   419   500   0  2006 ?        00:42:40 init [3]         
root         2     1  0     0     0   0  2006 ?        00:04:35 [ksoftirqd/0]
root         3     1  0     0     0   0  2006 ?        00:44:50 [events/0]
root         4     1  0     0     0   0  2006 ?        00:10:19 [khelper]
root         5     1  0     0     0   0  2006 ?        00:00:00 [kthread]
root         7     5  0     0     0   0  2006 ?        00:03:35 [kblockd/0]
root        54     5  0     0     0   0  2006 ?        00:00:00 [aio/0]
root        53     1  0     0     0   0  2006 ?        00:12:53 [kswapd0]
root       638     5  0     0     0   0  2006 ?        00:00:00 [kseriod]
root       669     1  0     0     0   0  2006 ?        00:49:43 [kjournald]
root      7228     1  0   404   684   0  2006 tty1     00:00:00 /sbin/agetty 38400 tty1 linux
root      7231     1  0   404   684   0  2006 tty2     00:00:00 /sbin/agetty 38400 tty2 linux
root      4020     1  0   820  2272   0  2008 ?        03:14:55 /usr/sbin/syslog-ng
root      1599     1  0   945  1020   0  2008 ?        00:00:00 /usr/sbin/saslauthd -a pam -n 3
root      1600  1599  0   945  1020   0  2008 ?        00:00:00 /usr/sbin/saslauthd -a pam -n 3
root      1601  1599  0   945  1020   0  2008 ?        00:00:00 /usr/sbin/saslauthd -a pam -n 3
root     32007     5  0     0     0   0  2009 ?        00:00:18 [pdflush]
root     19284     1  0   437   624   0  2009 ?        00:00:00 /sbin/udevd --daemon
root      6711     5  0     0     0   0  2009 ?        00:00:00 [kauditd]
root     18726     1  0   550   844   0  2009 ?        00:00:00 /usr/sbin/fcron -c /etc/fcron/fcron.conf
root      7784     5  0     0     0   0  2010 ?        00:00:07 [pdflush]
root     11131     1  0  1155  1848   0  2010 ?        00:00:22 /usr/sbin/sshd
root     11277     1  0  1373  1584   0  2010 ?        00:00:01 /usr/lib/postfix/master
postfix  11282 11277  0  1394  1668   0  2010 ?        00:00:00 qmgr -l -t fifo -u
root      6338 11131  0  1795  2200   0 Jan07 ?        00:00:05 sshd: root@pts/0 
root      6343  6338  0   762  1684   0 Jan07 pts/0    00:00:00 -bash
postfix  21039 11277  0  1380  1528   0 10:11 ?        00:00:00 pickup -l -t fifo -u
lighttpd 21132     1  0  1010  1380   0 11:15 ?        00:00:00 /usr/sbin/lighttpd -f /etc/lighttpd /lighttpd.conf
root     21165  6343  0   582   920   0 11:15 pts/0    00:00:00 ps -eF

lithium ~ # tune2fs -l /dev/hda3
tune2fs 1.41.3 (12-Oct-2008)
Filesystem volume name:   
Last mounted on:          
Filesystem UUID:          5384337f-6d08-4938-bc70-049a10dc3bee
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal filetype needs_recovery sparse_super large_file
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1994944
Block count:              3986128
Reserved block count:     199306
Free blocks:              2985298
Free inodes:              1708965
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16352
Inode blocks per group:   511
Filesystem created:       Sat Oct 23 11:45:48 2004
Last mount time:          Wed Feb  1 02:25:22 2006
Last write time:          Wed Feb  1 02:25:22 2006
Mount count:              31
Maximum mount count:      30
Last checked:             Sat Oct 23 11:45:48 2004
Check interval:           15552000 (6 months)
Next check after:         Thu Apr 21 11:45:48 2005
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           128
Journal inode:            8
First orphan inode:       164326
Default directory hash:   tea
Directory Hash Seed:      e397094b-92b8-4e0d-a0d5-9b9397d7bca6
Journal backup:           inode blocks

Work Load

Mail messages processed in 2010

lithium 2010 # for d in /var/log/2010/*; do echo -n "2010.$d = "; grep 'status=sent' $d/*/mail.log|wc -l; done
2010.01 = 53326
2010.02 = 48392
2010.03 = 65538
2010.04 = 77076
2010.05 = 34868
2010.06 = 214204
2010.07 = 247701
2010.08 = 377079
2010.09 = 407144
2010.10 = 212466
2010.11 = 1246
2010.12 = 727

Pages Served

Only the last few months were available in the logs, notice logs are not rotated :(

lithium ~ # grep 'Sep/2010' /var/log/lighttpd/access.log |wc -l
287049
lithium ~ # grep 'Oct/2010' /var/log/lighttpd/access.log |wc -l
294449
lithium ~ # grep 'Nov/2010' /var/log/lighttpd/access.log |wc -l
256327
lithium ~ # grep 'Dec/2010' /var/log/lighttpd/access.log |wc -l
260599

Google Mail Aggressive Spam Marking, Latency Issues in GMail

2011-02-15T01:17:00.000-08:00

Lately more an more of the legitimate message to our Google Apps have been getting tagged as SPAM. This is a big bad thing, not because SPAM is bad but because Google's Spam filter has been so good we (everyone) seems to forget to check the Spam label.

Symptoms of False Spam

Firstly these messages are shown with the following warning:

Warning: This message may not be from whom it claims to be. Beware of following any links in it or of providing the sender with any personal information.

More and more messages are being tagged, many are legitimate like ones from Chase, WellsFargo, LinkedIn, Facebook who have properly configured MX, SMTP and DNS/SPF systems.

Another part of the issue, which we can see from headers, are DNS errors which appear to be happening inside the Google network, vis:

Received: from mail-iy0-f197.google.com (mail-iy0-f197.google.com [209.85.210.197])
        by mx.google.com with ESMTPS id j8si5748968icp.124.2011.02.13.19.59.23
        (version=TLSv1/SSLv3 cipher=OTHER);
        Sun, 13 Feb 2011 19:59:23 -0800 (PST)
Received-SPF: error (google.com: error in processing during lookup of XXX+bncCOq0i6ayARCb0eLqBBoE1p7bnQ@edoceo.com: DNS timeout) client-ip=209.85.210.197;
Authentication-Results: mx.google.com; spf=temperror (google.com: error in processing during lookup of XXX+bncCOq0i6ayARCb0eLqBBoE1p7bnQ@edoceo.com: DNS timeout) smtp.mail=busby+bncCOq0i6ayARCb0eLqBBoE1p7bnQ@edoceo.com
Received: by iye7 with SMTP id 7sf5690121iye.0
        for ; Sun, 13 Feb 2011 19:59:23 -0800 (PST)
Received: by 10.231.39.205 with SMTP id h13mr719416ibe.4.1297655963810;
        Sun, 13 Feb 2011 19:59:23 -0800 (PST)

Intra Domain Issues

Another issue we've been seeing happen is completely internal to Google Apps. In this circumstance the intra-domain message of some Google Apps users have been rejected, with 500 level SMTP errors and/or some timeout issues.

We'll keep watching this one to see what happens, it's very intermittent at the present.

TDSS.tld4 - Microsoft Support Issues

2010-12-08T11:19:00.000-08:00

A client's computer was infected by this virus virus.
The symptoms were that visits to Microsoft and other security sites were blocked.
Many were re-directed to "get-search-results.com".

Microsoft offers free support for their systems when they are infected.
Just call: 1-866-727-2338

The issue appears to be a resurgence in the TDSS virus.
Windows update won't work, Windows Live OneCare won't work.
Microsoft Support won't work either, they simply ask you to run these tools that you can't get to because of the Virus. It's very much a chicken and egg issue, doctor heal thyself. If I cannot visit these sites how can I run these tools. I had to spend over 30 minutes on the phone with their "Technician" to explain that fact to them. What a waste of time.

Then, after re-trying all the obvious things that I already tried the "Technician" and wasting an additional 45 minutes. Says they don't know the answer (*shock!*) and will have to transfer me to an Internet Explorer specialist - then the call drops.

So I call back, get a new technician, waste more time getting this person up to speed. Then this tech transfers me to another "Technician" - that was more than 20 minutes on hold. This technician was supposed to help me get the IE fixed. This "Technican" then told me they could not help and send me back to the beginning of the line to PC Safety.

So back now to my fourth try with PC Safety support. By this time I'd already performed some clean up and got the system functioning much better. This guy says simply: Hey lets use Kaspersky tools. Shortly there-after we found the TDSS.tdl4 root-kit. Removed it, rebooted. Then use ComboFix, then reboot.
Should able to run Updates and other pieces successfully.

Ran into one more problem when the Updates failed, had to manually add some keys to the registry.
See our Windows Update notes for more info.

There was over an hour wasted with their "Support" factoring in support costs & the lost-productivity and this virus costs over $1000.

Keep your systems updated, get & use an Active Anti-Virus tool such as Kaspersky, use tools such as SpyBot and CCleaner to keep cruft from building up, where virii hide.

Google Apps Accounts Merging with Google Accounts

2010-11-21T00:30:00.000-08:00

Google has recently been deploying migration/integration of Google Apps Accounts to work more like a regular Google Account. Nice feature but migration is not as smooth as one would think.

I expected Google to implement a "seamless" migration. That is to say, if I had a Google Account of user@domain.tld and also Google Apps for domain.tld when Google Apps is expanded data from the general Google Account would be merged/migrated/re-associated - poof. Perhaps even the Google team could have created a conflict-merge tool. Complicated but, they have loads of cash and many intelligent people.

Actuality

For some of our clients this migration has been not a big deal, perhaps only one or two accounts were even affected. For them the case was easy - turn all features on and for the general Google Account they were simply deleted.

Other accounts needed to pull Calendar and other data to then re-publish in their newly-enabled full-featured Google Apps Account. This is a tedious process.

Delete your old general Google Account account by logging in, by-pass any migration steps and then pick "Manage My Account" and then "Edit" under My products section.

Migration Fails

Other users had some accounts that were tied to many, if not most, Google services such as Blogger, Checkout, Picasa, Analytics, Webmaster Tools, YouTube, &c. Here is a BIG ISSUE if you read this help page from Google it will tell you accounts connected to many of those services cannot be migrated - they will continue to work - and hopefully be migrated soon. It may make sense to hold off on migration until the process with these services is worked out.

rDNS Epic Failures

2010-08-06T10:16:00.000-07:00

Over the past week we've been helping a client get their systems properly configured. One of the issues we needed to resolve was reverse DNS entries (rDNS). Pretty simple task huh? Especially for the "network engineers" at a data centre. Surely they should know how to do this.

For more than eight days we had to work with this colo-provider. We asked for rDNS entries. After two days they responded, asking for clarification. Um? REVERSE DNS, add it to your NS!!.

A day later they responded that it was complete. We checked. No entry was found. SERVFAIL was the response from the `host` command.

Using `dig` directly to their NS we were able to see that their NS (the authoritative one for this block of IPs according to ARIN) was responding to these was REFUSED!!.

Usually `dig` responds with something like this:

;; HEADER opcode: QUERY, status: NOERROR, id: 64327
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

Their epic-failure response looks like this:

;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 34593
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

They are the authoritative NS for this block of IPs and they refused PTR queries from the public internet, which they are supposed to be serving. Apparently this had been an issue at their facility for weeks if not months. It took us three days to explain to their "engineers" why this was an issue! Read the fucking RFCs idiot!

Of particular humour was when they demonstrated to us their NS was working, by testing the NS from it-self (epic fail!). Our point was that the NS wasn't working from outside (London, Dallas, Seattle). What did they do? Test from localhost. Um? Are you kidding me?

It's unfortunate that there are so many amateurs. If you're sick of working with them call us. We follow standards and persistently push others to do the same.

Site Slowness? Too many Queries Maybe?

2010-08-01T23:09:00.000-07:00

We are currently migrating a clients site to a new platform. One of the original complaints with regard to the existing site was slowness. Think I found the answer today.

It takes 119 queries to display a product page. There was one query (exactly the same, no updates in between) executed 16 times! And another one run 30+ time because the author of the code didn't want to use the SQL operator IN or do a JOIN.

When you're writing a store front, ask your self. How often do you need to run

SELECT COUNT(*) FROM product_option where product_id = 345

We think 20 is too many.

Come on guys, trace your code and monitor SQL query logs when you're developing. These kinds of things are easy to catch and fix. Computer are fast but, lets not waste resources.

Keep your Systems Simple (kyss Method)

2010-07-30T16:18:00.000-07:00

We are helping a client assume ownership of a recent business they purchased.
We got all the code, domain, etc.
But! Like many IT projects the documentation was insufficient.

Our first step was to then audit the existing system. We discovered that multiple web-servers were involved, not just physical-servers, but Apache, Lighttpd, Tomcat, etc) all on the same box. Some Apache proxy to the Tomcat, etc. A complicated configuration with a wide set of dependencies.

It was clear there was some scrambling in the old project, code forks all over, duplicated libraries and a wide varsity of languages used (PHP, PERL, Java, JavaScript, BASH). There was duplicated logic-libraries in each language! More complications that introduce inconsistencies (ie: fix the PHP lib but forget the PERL - OOPS!). This is also clearly too complicated.

There are many reasons why projects get to this state. Maybe one engineer likes Java more than PERL or historically it started with a shell-script. History, politics and engineer preferences can all be a factor. At the end however, we don't care why.

Complexity Kills. Computers already have 1000s of layers of software between the Human and the Hardware. Putting a Rube Goldberg application on top does nothing to help.

Our KYSS Method

So what did we do to fix it? Our policy was to pick the low hanging fruit first. About 80% of the project was PHP we decided to push it all there. We simplified the system by replacing the Tomcat/Java piece with a PHP component. Reduced by one language and Tomcat dependency. Then we focused on performance of that component to reduce resource (network/memory) consumption by over 60%) Added a bit of caching and it was done. Then the back-end libs in PERL and Bash were streamlined to PHP as well.

The end result was taking a project with three web-server-software dependencies and reducing that to two (Apache/Lighttpd). Reduced the language count from five to two. A result of those to is a smaller, faster and simpler configuration that is much easier to maintain.

It will take roughly eight months to return on our investment of cleaning and simplifying this project. Awesome.

There are many cases when multiple languages need to be used, too numerous to list. In the end a seasoned architect & business analyst need to evaluate this and make a plan. Otherwise, like in this example, developers/engineers go their own fractured way and entropy reigns.

PHP Timezone Handling

2010-07-24T19:14:00.000-07:00

Today we saw another abomination of Timezone handling in PHP.
100s of lines of code were used with many hard coded values and the system was still incomplete.
So, in efforts to put a stop to this we've posted our implementation of php timezone handling which we think is easy, fast and portable.
We've included a few other examples too, but just to show why we don't use them.

javascript:void(0)

E-Commerce Migration from Django/Satchmo to PHP/Radix

2010-07-16T16:45:00.000-07:00

One of our clients is using an abomination of e-commerce called Satchmo.
The client has described the solution as "cumbersome", "difficult", "slow" and a few other words not suitable for print.
We've been assigned the task of re-vamping their solution.

Despite the odd database schema and lack of documentation we're confident that this can be re-worked into our PHP/Radix based solution in short order.
Check back in three weeks for details.

Edoceo's Blog

The Trouble with NetSol

How to Fix It

Moving Mail via IMAP from Account to Another

List the Source Folders/Labels

Copy to Local Directory

Copy one IMAP Account to Another

A Week without Flash

Adobe Flash Fail Sites

Important to Remove

Getting Started with PhoneGap on Linux without Eclipse

Install the Android SDK

Install PhoneGap

Building the Sample

Running the Sample

See Also

Credit Card Brand Logos as CSS Sprite

Network Solutions DNS Migration - Caution Parking!

McAfee "Secure" / HackerSafe - Bad Config?

Stop Wasting Time with Social Media

Credit Card fraud and New Egg.com - Update

Single Use Credit Cards / Virtual Account Numbers

Credit Card fraud and New Egg.com

Google Apps Transition: Checkout

Google Apps Transition - Google Analytics

See Also

Google Apps Accounts Migration for Blogger & Checkout

Migrating Blogger or Checkout with Google Apps

See Also

Google Apps to Accounts Transition Failures

Checkout Fails to Migrate

Analytics, AdSense and AdWords Fail to Migrate

Blogger Fails to Migrate

Alerts Fail to Migrate

Google Apps, Talk / XMPP, Andriod Failures - Part 2

Update from Google

Update from Edoceo

Google Apps, Talk / XMPP, Andriod Failures

Disabling Google Talk / XMPP

Attempts a Resolution

How to Fix (what Google should do)

Update from Google #1

Update from Google #2

Update from Edoceo #1

Our Best Uptime - Ever!

The Hardware

Software

The Uptime Proof

Work Load

Pages Served

Google Mail Aggressive Spam Marking, Latency Issues in GMail

Symptoms of False Spam

Intra Domain Issues

TDSS.tld4 - Microsoft Support Issues

Google Apps Accounts Merging with Google Accounts

Actuality

Migration Fails

rDNS Epic Failures

Site Slowness? Too many Queries Maybe?

Keep your Systems Simple (kyss Method)

Our KYSS Method

PHP Timezone Handling

E-Commerce Migration from Django/Satchmo to PHP/Radix