Last night, a horrific attack took place. Neighbours had called police to report screams and loud banging coming from the flat above. Sadly, when the authorities arrived, there was no screaming. There was no banging, only silence.

It was too late.

When police entered the flat, they found a woman in her late teens beaten and bloodied. She laid lifelessly on the floor in the kitchen.

It was too late. She was dead.

This wasn’t the first time someone had called to report a violent incident. The woman’s post mortem indicated numerous injuries, sustained from years of abuse — broken bones that didn’t quite heal straight, unhealed fractures, sprains, etc.

A look at the girl’s medical history showed numerous trips to the emergency department for various cuts, bruises, fractures, and internal organ damage. To make matters worse, she was pregnant.

You see, when police arrived, they found a piece of paper with a hastily scrawled address on it.

It was the address of a nearby women’s refuge. She was trying to get out, but was too late.

The man responsible for her murder, her partner, is thankfully in custody. He was a paranoid, unemployed IT worker who was, according to the police report, intoxicated at the time.

When the police questioned him, the man admitted he knew she was about to make a run for it. How did he know she would run? Easy. He regularly monitors her Internet usage and saw a Google query for the local women’s refuge. He confronted her and lost his temper.

He didn’t mean to kill her, he said.

She would have likely made it to that shelter if she had used https://www.google.com. Think about that when you’re moaning about not being able to see referral data.

The above story is fiction, but it is based on at least two real cases I know of. Many people seem to think the removal of the referral data was meant to aid privacy. In reality, it’s an unfortunate side product, albeit one that Google seems happy to promote as a feature instead of a bug.

The very genuine reason that moving to HTTPS is an improvement of privacy is to stop people from accessing query data and the results of queries on networks, between the user and Google Servers. Around the world, this will help increase access to information and provide some level of protection.

Overall, this is a positive step and one that Google has taken prior to having it enforced upon them by authorities. Many, including myself, believe it’s something they should have done 5 years ago and are already to late. For others, it’s throwing the baby out with the bath water and the referral data issue should be solved before enforcing https. Of course, a third group just thinks Google is a corporation and it can do what it wants. They are probably right.

In the past, people have suggested it should be opt-in, but in my fictitious story, the girl would not have known to opt-in. She certainly wasn’t going to ask her partner how.

It’s not the only situation where this comes into play:

• Someone seeking news of their brother or sister in Iran
• Soldiers in a war zone seeking more information about a natural disaster that hit their hometown
• Finding information and locations people can get help during genocides or persecution
• Even SEOs looking for donkey porn

I, like most people, want Google to continue to provide referral data and I hope to see the system they are testing with their AdWords customer rolled out, even if it means passing a UTM string of their choosing instead of mine. But I agree with them: rolling out https as a standard it’s worth it. Even if it saves just one life. EVEN if it causes you and I some inconvenience.

Updates
After sending this to a couple of friends to proof read, a couple of queries came back:

1. Wouldn’t he have seen q= in the query string – Initially I was under the impression no, as Google would realise this, and under https, use POST rather than use GET. It turns out that “encrypted.google.com” uses GET. However, reports say Google is rolling out POST on https search requests, and I assume these will become the default
2. Wouldn’t adding additional query string parameters cause issues for sites? Yep, that’s probably why Google hasn’t rolled out tracking to normal search results.
3. From the comments, They are deliberately dropping referrer on SSL as well? This is not the case on encrypted.google.com but some reports from Google “never speaking officially” Matt Cutts implied the new system would. Unofficial comments I’ve had is that it’s not the case but something that is a “potential” in the future (presumably once a better solution for current issues are found)

A thought did pop into my mind: what if Google also introduced https for cached results? Sending the data via a post request. How would authoritarian regimes react? I’m guessing badly, so perhaps this will be the start of a truly fragmented web.

However, what really interested me were their new identity services. Now, several sites have been jumping up and down with headlines like “PayPal becomes an identity provider,” which is a tad odd. They have been for a while with both an Authentication API (Though, I think Coding Futures may have been the only company outside of PayPal who were and still are using it.) and more recently, OpenID implementation. What the new services do is bring these experiments to the main stream, with support for both identification and authorisation.

PayPal Identity Implementation

The OpenID system is PayPal’s new standard mechanism for identification for third parties. It’s part of a collective of mechanisms for identity provision called PayPal Access. This is an interesting step to create a brand.

One of the biggest problems we have faced in the past with PayPal identity services is convincing users to make use of it. Many users, when asked to “Login” via PayPal assume they are being asked to enter a payment flow. The OpenID implementation follows the OpenID v2 specification. It can return basic details such as name and the user’s address, and for people running PayPal related services, two exceptionally useful parameters: PayPal account type and the PayPal Verified status. Unlike a normal OpenId implementation, the domain you’re using it on needs to be whitelisted. And to make full use of the system, the entire process should be under https.

PayPal Authorisation Implementation

What is new is the announcement that at last PayPal will be supporting OAuth. OAuth is a token-based service that, once a users identity has been confirmed, grants permissions for your site to take certain actions. Initially, these actions will be limited to accessing certain information, but hopefully, the old permissions API and Adaptive Payment flows will be initiated from Oauth token.

PayPal Access – Branding Identity

As I already mentioned, for me the biggest issue with PayPal’s previous identity services was the lack of branding and PayPal’s attempt to instill user confidence. Having spent so much time and energy into preventing phishing scams, PayPal trained its users not to trust anything that looks like PayPal from a third party site. Consequently, digital payments, embedded goods payment flow and its previous identity service attempts have had issues with consumer confidence.For PayPal to be able to push PayPal Access, they will need to not only push the concept to developers, but also to consumers.

Unifying the technologies into a brand is a good starting point. Making the user experience and language coherent and obvious that no additional charges will be made isn’t quite there yet. The PayPal Access login and approval screens still look too much like a payment flow. They also fail to tell the user why they are logging in until post-login. That said, as PayPal access becomes more widespread, and people are regularly confronted with it, confidence will improve.

On to the cool stuff…
If you are a privacy advocate do yourself a favour look away now…

User Profiling with PayPal Identity Services

In addition to the refreshed OpenID and implementation of Oauth, PayPal also added some additional identity services to improve user profiling: Prospect API, Segmentation API and Product Recommendation Service API. Each of these allows a site to gain information about a PayPal user and their buying habits. It also harnesses the power of the Intelligence Engine on Ebay to product categorise — You can cross-sell from your own inventory to that user.

Prospects and Segmentation APIs

How much is any given user worth? What are their spending habits? How active a shopper are they? With the segmentation and prospects API, a merchant is able to profile individual users overall PayPal habits, including how frequently they shopping, the average spend value of a shop, etc. None of this is finite data. Instead, users are grouped in terms of usage frequency the groups are: Engaged, Habituated & Casual

So, what can you do with this data? Well, for your initial sale, probably not much. But post-sale, this data provides extra details about the user’s sales prospect. For example, if we wished to sell a group of products, we could do so in two ways: individually at a low price, or bundled together at a higher price. With our customers profiled we could target groups differently.

The casual user but big spending group could be targeted with the bundle deal. The super engaged but low spending users could then be targeted with individual products over a range of time.

This does lead to the obvious privacy concerns, of course. In reality, people have been buying and selling this data for years, and all major eccomerce sites are using prospect analysis of some sort.

PayPal is bringing this down to mid-level merchants. It’s worth emphasising that this is PayPal, so expect it to be near impossible to get access to these APIs, without jumping through a dozen or so hoops, while standing on one leg.

The problem comes in when offering these services — There really is no way to do it, without providing user information. The grading bands are wide enough that the demographic information could not be used to judge any financial information. After all, just because someone is a heavy PayPal user does not mean they’re wealthy or poor.

The new data, while interesting and useful, really is scratching the surface. An aspect I would love to see included in the data is a user’s refund/chargeback rate and their average subscription rate. For membership sites and other recurring subscription sites, an idea of how long they have got the user for could totally change the way they present information.

For example, if I know a member subscribes for roughly 3 months, while my overall average is 4 months, I can change my content delivery strategy, so that open content (that starts 1 week and ends the next) is sitting over a 3 month period and not the 4 month mark.

Future of PayPal Identity Services

The announcements are a good start, albeit a slow one. Most of what I have talked about is still in Alpha (with exception of OpenID) rushed out for Innovate. And given PayPal’s track record, these are still a year or more away from becoming a reality.

The shift to OAuth is important. PayPal is not just providing services to identify users, but also a fully-fledged authorization system, which is an industry standard. This has to be applauded, but the old user experience bugs from the old identity system are still there. Even with branding and without a large marketing push, consumers will still struggle to see PayPal as being used for anything other then paying.

One of the things I hope OAuth brings is a unified system. Then, I as a merchant can use OAuth to authenticate and authorize the user, make a call to the segmentation API, present my offer, and start the payment flow with the auth token. This way, the user only has to sign in once.

As a merchant and developer, I want a flow similar to:

Login -> Segmentation API -> Show Offer -> Make Payment -> Show Upsell -> Setup Subscription

This flow should be easy for the consumer. Sadly, this flow currently requires 3 separate PayPal sign ins. And each successive sign in degrades consumer confidence and will power.

With the whole X.Commerce brand, PayPal is trying to create a platform like Amazon and Facebook have. The cornerstone of any platform is opening it to third parties, but given the nature of PayPal’s business, this has to be done in a controlled manner.

The identity services are a step in the right direction, but so often before PayPal has taken a step in the right direction, only to have its own bureaucracy prevent any real usage. I really hope this is not going to be the case and the X.Commerce platform lives up to its promise.

Trust me some people are intent on spamming donkey porn to the world!

Fear not if you do not fancy your site being used to spread mule based filth with good security practices and some simple hardening of your site.

OxOx Creative Commons

The advice given at Glyns talk covered 99% of what is required but at a whirlwind pace, if people are interested in learning more Coding Futures runs half day workshops on “brand security” for digital agencies these focus primarily on WordPress and a lesser extent general good practices in social media and brand technologies such as Twitter.

The workshop is £185+VAT and we have a few spaces available for the September workshop. For more information please check our the Brand Security Workshop site.

The workshop cover 3 key areas of brand security;

• Prevention
• Detection
• Reaction.

Using a mixture of hands on examples, case studies the workshop will focus on two of the most common platforms used by digital agencies, WordPress and Twitter. The workshop will focus on understanding threat models, hands on protection for WordPress and Twitter accounts, tools to aid in detection of hacks and perhaps most importantly dealing with the aftermath of a hacking attack to minimise damage not only on the compromised site but other accounts effected.

The Workshop is a hands on event and attendees are encouraged to work on sites. As such it is suited to people who have control of their or their clients sites (If you have FTP and WordPress admin details). No technical expertise is required though an understanding of HTML and WordPress will be advantageous to get the most from this workshop.

By the end of the seminar attendees should have a more complete understanding of WordPress security with practical advice for their own sites and a greater understanding of the Twitter platform and best security practices that can transcend social media platforms.

Basically don’t fancy donkey porn on your clients sites? might be worth coming along for more information and to register please visit the Brand Security Workshop site

As the Technical Director at Coding Futures, I now have a day, night and weekend job, which simply takes the majority of my time. I wouldn’t change it for the world, but sadly, the blog has been the neglected child. That needs to change, so I’ve made some changes.

First, even though it’s been up a month: SURPRISE! The blog has had a theme makeover with the help of my friend Kat Neville, who designed the theme nearly 2 years ago. I just finally got around to getting it up and going.

Secondly, while I was offering consultancy through the site, I was never comfortable having advertisements on it. With consultancy gone, I’m going to introduce some advertising. It will also give me a good excuse to play with adApe (ooo I can do product placement to) before it launches. If you’re interested in advertising on the site, please do get in touch.

XKCD 870

As this is timnash.co.uk, advertising will be slightly different and experimental. Therefore, advertisers will have the ability to run mini retargetting campaigns on the site. If you are concerned about privacy, the good news is the retargetting will obey do-not-track headers. Over the next couple of months, expect new and interesting advertising experiments, which hopefully will be good exposure for advertisers and of interest to readers.

For me, the way to get back into blogging is to make timnash.co.uk exciting again (and force me to give it the time it deserves). I’m also interested in hearing what people want me to write about.

Here are some of the drafts I currently have. Which should I finish first?

• Long Landing Pages, Success a myth or a fact?
• Determining Connections within networks
• My Super Affiliates help me screw you!
• Who owns a site’s analytical data
• Digital Suppression a slippery slope to reputation management
• Identifying Users, no such thing as anonymous
• Expanding your retargetting methods
• Something completely different

Not all those posts will be published. Some are just a few words, while others almost finished. So, what appeals to you?

Remember, if your interested in advertising, take a look at what timnash.co.uk can offer.

In July, the BBC launched the Dr Who streaming service. For a few Facebook credits, non-UK residents can watch a select set of Dr Who episodes for up to 72 hours. (This is just one example of how the garden is changing.)

When the Beeb launched their service, Twitter and Facebook exploded. Why? It’s the BBC. Naturally. everything they touch has to be investigated in minute detail, discussed on social networks and in the wider media here in the UK. It’s also a change of direction for the BBC, which is now relying on a 3rd party. Indeed, it opened a debate on should the BBC be in part paying a third party to put content in a closed garden?

Over the weekend of the launch, I had to stifle laughs. Friends were explaining how much it cost, how they wouldn’t make their money back, and most importantly, why the BBC would give 30% of money to Facebook!

You see, as the technical director and part of the team at Coding Futures, I’ve helped several major clients launch large Facebook applications. We had also been working on a way to take that knowledge to put it into our retail software, Your Members.

The Your Members WordPress membership plugin leans heavily on developments coming from our work with enterprise clients. Over the last few months, we have been building a feature-rich Facebook addon that allows you to create applications, protect content and pay for it with multiple gateways, including Facebook credits while in Facebook.

What’s more, the same software already has a streaming addon, which allows you to securely stream video from Amazon s3 via progressive download, or create an Amazon Cloudfront distribution and do real RTMP streaming.

Now, you would expect the Beeb to use it’s massive resources to hire a development team like Coding Futures to create something unique. They didn’t. In fact, they could have created the exact same thing for less than $200 using Your Members. Which, while great they have helped prove a concept, have they really pushed the boundaries?

Now, basically anyone could create their own pay-to-view Facebook video service. If you want to give it a go, here is a step-by-step tutorial for mimicking the BBC Facebook application in Your Members.

Of course, to use Facebook, you don’t have to pay them 30% of earnings. That’s only when using Facebook credits. You don’t need to take payments at all, and for the moment, only “games” are required to sell via credits, so other applications can use different payment methods.

Which leads to one of the more interesting concepts to developing within Facebook: the “likewall”. Just like a paywall, content is protected, but it only costs the user a single click to access content rather than a financial payment. For digital brands and ad agencies, the idea of a likewall is only just taking off mainstream, and it will be interesting to see how Facebook deals with incentivize likes. Currently, their terms and conditions on the subject are woolly at best.

So, the obvious question is where to next for the BBC? Is this first foray their last? Or, will we one day see iPlayer inside Facebook? (For non-UK peeps, this isn’t some strange apple device you haven’t heard of, but rather a BBC streaming service.) Probably not. If only because the people in the BBC working with Facebook are not the corporation we know here in the UK. They belong to BBC Worldwide, which is basically the BBC’s non-UK commercial arm.

So, if not iPlayer, more classic streaming episodes, perhaps. Certainly, if you can get over that 30%, using both Facebook Credits and another payment gateway could be one way to save costs (plug: Your Members can do that ), and in turn, make this an effective medium for any company with media content.

Another interesting possibility is organisations like Open University putting their online course material on Facebook for their students. And not just big organisations. Suddenly, anyone running an online course could have them available to individual users within Facebook. How about an online course on Facebook in Facebook, very meta!

For many people today, Facebook is the web. It’s application platform looks ready for prime time, and Facebook Credits are starting to be used regularly, even by non-farming related gamers. Is this the time Facebook marketing really grows up? Is it possible to run a Facebook-only business?

Should the BBC put money into a closed garden like Facebook, given that large companies owning closed gardens tend to start imposing more restrictive rules (cough apple)? Can any business truly survive by operating solely in Facebook? While it’s certainly technically possible, do you think it’s feasible from a business perspective?

I think it is, but having watched businesses been burned by similar platforms (Twitter/Apple/Microsoft) suddenly changing the rules, it’s not there yet. I think, until the platform fully matures, people will continue to make small steps and build small corners of their sites into Facebook. And now that it has become possible to do so easily, do agencies and businesses really have an excuse to not take those steps?

Disclaimer: I work for Coding Futures, the developers of Your Members.

Image: http://www.flickr.com/photos/ingythewingy/5178899845/

East Coast Rail and Failing credit cards

Like many northerners I regularly travel down south and prefer to take the train from Leeds the only real choice is East Coast Main Line, gone are the days of GNER and instead we are stuck with a nasty money pinching publicly owned disaster but this is not a rant about no more free wifi (or how you can just swap your mac address to get access) or cutting stations from the map but about their website.

East coast website is the normal place to buy advance tickets and it offers up to 20% discount on advance fares which are the only way to travel at a vaguely reasonable price. Recently at Coding Futures we have had a nightmare of a time booking tickets today I decided to experiment and work out why.

Card Declined for no reason?

We would go through the ticket flow something we have done a hundred times and put in all our details, as always the billing address details would be empty (though the card number and other bits were there) and we would populate with the office address, go through the dreaded and totally insecure 3d secure and come back to be told there was an error. A few hours later the charge will appear in the bank and within a day the bank will reverse it.

Several other cards and all the same thing, so what’s the problem?

Incorrect Billing address

In frustration I contacted their web support team I’m sure it was on the paper infront of him but the poor guy was utterly illogical telling me I had used two cards my bank was linking the two cards and using the wrong address. Strangely enough my bank has not linked my personal and corporate card and it was utter tosh but it did spark an idea.

The account I had been using, was my existing account on East Coast which had in the past had my home address what if it wasn’t my bank but East Coast who was linking the cards, easy to test so I quickly created a new account added the corporate card and as if by magic it worked. So that could be a fluke so I added my personal card (with a different address) filled in the address as normal guess what came back declined.

It would appear to not be the banks linking cards but that East Coast is ignoring what Address you have been putting in during the booking process and taking the one from the first booking of your account…

Intuitive nope, but if you have been having the same issues the solution is to create a new account for your card with your new address, while we have written and suggested strongly to East Coast the problem exists I don’t imagine it to be fixed anytime soon.

Dear PayPal

I think we are petty close, I develop on your services, you charge me lots of money in fees, I go to your conferences and generally moan to your dev team. Normally I keep my scorn of idiotic mistakes to just your ears but PayPal Pro Hosted really is a nasty mess.

So what, most of PayPal is a hideous mess! But the Adaptive APIs are so nice and a standard you should be looking to (though docs still need updating and the SetPay API is just nuts). Hosted Pro on the other hand is like someone took a bunch of kindergarden kids and asked them what they want in a system result shiny encrypt-able buttons.

The thing is it was so easy for it not to be…

1. User comes to site and starts payment flow
2. Server sends some details to PayPal, PayPal returns a token, which you append to a URL and open in an Iframe.
3. A box with Credit Card details appears you fill in the card details and hit go, on success you return the frame to a dedicated page, and on failure another if you want you can even frame bust on the way out.
4. The server then can double check via getTransactionDetails with the token you gave.

Sound familiar? It’s the same way your other APIs work, and guess what it works quite well!

It’s neither hard nor complicated, it doesn’t involve pretty buttons, and if someone wants to send extra parameters across they can do so in a controlled sane manner.

This seems like such a sensible approach, it’s hard to believe it wasn’t thought about and unless I have missed some major flaw, the current implementation still generates and populates an Iframe it just does it in a really ugly manner especially when you switch javascript off. The worries about security with Iframes are still there but generally with a far worse user experience, as in most cases Hosted Pro requires at least 1 more superfluous button and a lot of styling to make the Iframe work in the page.

Right now PayPal Pro Hosted is not a solution it’s a toy, I can sort of understand why it’s perhaps a little unloved, but when I saw it originally I really wanted it to be so much more I wanted a feature rich chromeless PayPal screen which I could take credit and debit card transactions, I wanted reoccurring billing and the option to authorise cards. Just think if you actually put the time and effort you did into Adaptive into PayPal Hosted it could be an awesome service and let’s face it with PRO you are not only getting the transaction fees but also charging a monthly fee so its in your interest to make me want to use it (ok not me but my clients certainly).

Sadly I’m having to look for an alternative for our in house apps, as I do not want to go through running a full PayPal Website Pro integration, especially when sadly there are cheaper merchant gateways out there that won’t suddenly freeze access to all your account and money. I’m still a fan, I’m not even that upset I’m just disappointed…

Hugs and Kisses

Tim

If anyone has a suggestion for a UK based solution, that uses Iframes to process the actual payment, rather then web forwarding, or doing Direct Card Payments, please do let me know.
Hopefully the above letter can be used inside PayPal to chivy some sort of sanity amongst them, but meh I’m just a pesky developer.

Photo by P_A_H via Flickr

So, here are some of my favourite tips when dealing with mail lists segmentation you may not have thought about:

Recently purchased list

“Amazing savings on TVs get a 42” for just £399”

Such an awesome offer, but I just bought a TV, which is why I’m on this list!

Sods law says an amazing sale will always happen after you purchase, but one of the most prolific reasons for unsubscribes with new users (and to a lesser extent existing companies) is badly timed emails. Keeping track of people who have recently purchased, and making sure they are not sent “sales emails” except for specific up-sells, will help your readers avoid seeing you as insensitive money grabbers. It will reduce your costs, and increase conversion rates, by reducing one of the groups least likely to purchase. Conversely, having a recently purchased list also means you can use this list to flog direct product up-sells. “Need help setting up that TV?”

Segment by purchase

You know who has purchased what, so make sure your email list knows too!

Segmenting by purchase has 2 very important uses: 1) You won’t try to sell someone something they already own 2) You can sell them something they are going to need or might not own. Segmenting by pages they visited when logged in may seem like an exceptionally cunning plan ala Amazon, especially for more niche sites, where it does work…

“Today only: One day sale on Your Secure Stream “

But, if you are selling lots of products, chances are the user is just browsing. By segmenting by purchases, at the very least, you can compare that to their behavioural history, before suggesting rattles.

Behavioural History

I have a great tip for those looking to show a marked increase in conversion: don’t send emails to people who won’t purchase. Simples!

Virtually all good mail campaign providers will provide statistics and allow you to segment based on who clicked what, who opened what, etc. If a user rarely opens emails, don’t send him daily deals, as the next one they open will most likely lead to an unsubscribe.

Instead, target him only on major campaign pushes. Likewise, the person who opens regularly, but rarely clicks is a window shopper. Entice them and target them with coupons. Make offers especially relevant to them. You may also want to use this group for surveys and feedback. Chances are they will click a “free” anything.

Customers Mood

Let’s face it: things happen, mistakes are made, and stuff get’s broken. It’s what customer support is for, but sometimes, it might not be wise to push the latest awesome deals to an upset customer. That said, one of my more interesting emails recently was from a company, who sent me an email suggesting I upgrade to their premium service, after providing some appalling customer service. I wasn’t sure if I should laugh or cry; it was either amazing timing, or a deliberate attempt at an up sell.

Looking at customers moods may not be a metric you have on hand. If it’s not, it might be worth asking support staff to try and gauge the mood of a customer through the support cycle.

I tend to recommend segmenting at least by who has contacted support recently, and if possible, their last known mood. Generally, we will not contact people who have requested recent support,  unless the outcome was positive.

Building segments in MailChimp

In theory, you could do the above by hand, assuming you had the metrics to hand, but it would be long process, and one most mail manager GUIs of are not suited for. To get people started, here are some handy hints for doing the above in Mail Chimp. (Other mail managers can do the same, and if you are dealing with lots of products, you may wish to do this in house, as it will get messy.)

I’m going to assume you have built yourself a handy class for handling mail, which has some nice separation of method, parameters and all the boring stuff.

Setting up segments
To add a new segment within the API use the:

listStaticSegmentAdd (APIkey, ListID, Name)

The name can be more or less anything, but remember that it’s what you will see in the Mail Chimp screen. On success, you will be returned an ID. This ID is relative to the list. You will therefore need to make static segments per list.

Removing an unneeded segment is just:

listStaticSegmentDel (APIkey, ListID, seg_id)

Where seg_id is the relative segment ID. Note: It has to be the id, not the name.

Adding users to segments
This again is pretty straight forward:

listStaticSegmentMembersAdd(APIkey, ListID, seg_id,batch)

Batch is an array of IDs (of users in the system), or email addresses. You can mix between the two, if you want to be crazy. I tend to find using emails easier, as I’m often adding data from my own systems and won’t be using Mail Chimp’s internal system. Note: this does NOT add people to lists. If they are not on the list, it will return an error for the given user.

So, can you guess how to remove users?

listStaticSegmentMembersDel(APIkey, ListID, seg_id,batch)

The nice people at Mail Chimp were not imaginative in naming conventions, which to API writers is a good thing. Again, it’s worth noting the Batch is an array of emails or Ids. If you want to wipe all the users from a segment:

listStaticSegmentReset( APIkey, ListID, seg_id)

As tempting as it is when doing temporary segment lists such as new purchases etc to wipe the whole list on each cron job and rebuild, you may wish to avoid it. While it’s nicely efficient from your side of things, it may potentially upset Mail Chimp. Though, this would depend on the size of your list. Also, I confess that I am using this bad practice on one of my lists with a segment that is wiped and rebuilt weekly.

Segmenting by Lists in Mail Chimp
I’m sure it was the middle of last year when myself and Illiya Vjestica MD from Smart Dog were chatting via Twitter. At the time, I had written a simple tool for doing segmentation by list, so you could select a list to segment another list by, when someone from Mail Chimp mentioned the feature already existed. I even remember looking and going, ‘oh yeah, so it does’. Well, I can’t find it any more, and nor can the people at Mail Chimp live support. And, with their URLs all screwed (hmmm there is an SEO/usability post in there somewhere), I can’t even check properly.

That said, it is fairly easy to do via the API (though, sadly it means you will need to run a script before going into Mail Chimp).

1. Add a static segment on your main list, or Sorry Mail Chimp, listStaticSegmentReset if one exists, especially if lists are regularly updated.
2. Use listMembers to get all the user Ids from the list you are segmenting by
3. listStaticSegmentMembersAdd using the list of Ids you just did against your main segment.

That’s it!  The only pain is needing to run it before you go into Mail Chimp.

Other ideas?

This is the point where I go over to you. I haven’t covered bits like geography, time, etc. So, what are some of the ways you segment your list?

I’m currently midway through a major behavioural modelling project. It’s complete with large scale re-targeting, both within the site and via advertising networks, as well as using techniques such as the CSS history hack to collect data about whether a visitor has visited our competitors.

Indeed. Looking at the sort of stats we are collecting:

Everyone coming into one of 3 sites is being tagged, using a browser fingerprint. It’s similar to the level Panopticlick project uses in their efforts to educate users on how to be on the safe side. We also use a persistent storage in the form of persist.js to “cookie” the visitor. Lastly, any third party software, which is capable of accepting custom values, has the user hash added, making tracking a user across the board as easy as possible.

To put this in perspective we can at 1 click retrieve:

• If the user has purchased
• How they arrived on the site
• A rough idea of age
• Rough idea gender
• Where they are coming from (not just country but are they at home or work)
• If they have visited our competitors
• What pages they have visited
• What offers enticed them
• If reinforcement marketing is working
• Any lead mechanisms (email/twitter etc) we may have them subcribed to
• If they are part of a focus or test group
• What ad group they arrived in
• Which split tests they have been set up with
• Where they clicked on a page and when

Basically, everything they have done on the site to the tiniest detail can be looked at, analysed and dissected. What’s more, the average user will not have a clue. That list would terrify many. I mean, if little peeps like us are doing it, then imagine what people like Google are doing. Best get your foil hats now!

Privacy at heart of Behavioural driven campaigns

One of the things that has been important for us from the start of the campaign is for our visitors to be in control, well a bit anyway. If possible, we want them to be able to opt out of our orwellian vision. The problem is how?

Removing data

Let’s assume we have a user who does want to opt out. The first stage is to remove their data. Since our system has a database, this is fairly simple. Just delete their row in the visitors table and any associated data in the meta table. Small snag: this doesn’t remove the data in third party applications and causes data corruption in the master table. Really, there is not much we can do about the 3rd party applications. Where possible, you can try to automate them, but normally the only option you are left with is giving a user a link to the application’s opt out procedures, if indeed they have one at all!

With your own applications, we have gone down the route of what we term “anonymous annihilation”. All our users are split into testing groups, and the user’s information is overwritten by an average of all those in the test group. The only data we keep exact is country. The IP is overwritten to 999.999.999.999, which makes an easy way for us to exclude the data in reporting, and their user agent finger print is reduced by us removing all the plugin data. Suddenly, we can’t tell them from Adam, except for that Persistent “cookie”, which actually is quite a pain to remove. But hey ho! That was the point. The issue is how do we not track them in the future?

Cooking the excluded

The only real way to exclude someone from an opt out system is to know they have opt’d out! But, to know they have opt’d out, we need to either maintain some information, or tag them in some way. Neither of these options are very palatable to the end user, but ultimately, at the moment, it is the only real solution. When opting out, I suggest using a traditional cookie rather then a persistent storage, clearly named within the cookie and make it clear this is what you have done. The downside, if they clear their cookies and come back, you generate a new profile and the circle starts again. But hey, you tried!

Looking to the future

Right now, there is a lot of talk about “Do not track” methods, especially amongst browser manufacturers. Google is releasing a new extension to allow you to prevent tracking (The irony sure will not be lost on them) and there is a more public discussion from the Mozilla team. Both seem to be heading down the route of the browser making the decision to prevent storage, which is great in principal, but has 2 major obstacles to overcome:

Persistent storage is all about hiding things in the most obscure places such as flash storage, where browsers do not have control. Therefore, simply assuming the browser is in control of all storage would be a mistake.

Carpet banning of data, would be frustrating and would effectively break a lot of the modern web. Cookies and storage are used in every aspect of web development, from ad tracking through to analytics, to storing shopping carts, to changing the colour of a site. Users are not going to want to be prompted every time, so they are likely to adopt an on or off approach.

One of the more interesting and hopeful projects is the idea of using headers. Proposed by Mozilla, the idea is that the client browser sends a HTTP header to the server, telling the server the user does not want to be tracked.
X-Tracking-Choice: do-not-track
It’s then up to the server to determine how to handle this. I think this is a great step forward with one major addition.

Telling a browser to send the header, I would like to see a method that allows sites to instruct a browser to send the do not track header. In effect, when someone clicks opt out, the site tells the browser the user wishes to opt out. Now, obviously, you don’t want a site to be able to opt people in, so the mechanism should be one way, and not mandatory for the browser (i.e it shouldn’t override an existing user preference).

The mechanism I propose has one major issue, at the start I explained this was a multi site campaign, but the mechanism is for only one site, and I can’t see a safe way around.

What do you think? Should we adopt Do not click header? What about the ability for a site to ask a browser to enforce it? Would other advertisers use it?

Now, I use a large screen and I have noticed that I am using the previews and reading from them, so maybe I was wrong. Certainly one thing that will change is that people will need to start thinking about how their site looks in the new Google Preview.

The obvious question… What browser is Googlebot and what render engine is it using?
Well, until now, we have had to make do with what our own stats tell us… No longer….

We can now see what Google see’s. For example, we now know certain full screen takeovers in JavaScript are being rendered. This possibly indicates Google’s ability to deal with JavaScript on page is far greater than people initially thought, but we can ask Google a far simpler question:

http://www.google.co.uk/search?q=what+browser+am+I+using

Can you guess what happens next?

Or, if you actually save the image:

From the screenshot, if you squint, you can see a full useragent string, the platform, and some basic capabilities. You may wish to note the cookies option for the spammers amongst you:

Cookie Enabled: true;
Language: EN
Productsub: 20030107
appcodename: Mozilla
Vendor: Apple Computers Inc
Platform: Linux x86_64
appname: Netscape
useragent: Mozzila5.0(en_us)applewebkit525.13(khtml,gecko)version3.1(safari521.5)

That was just from asking 1 question, and picking 1 result from the Google results. The obvious thing is to make a page that tracks the information about an arriving user and collect as much information as you want. Then, add a simple search phrase and point some links. On the page, make sure you have a nice clear reference number.
Wait for Google to crawl, look up the reference number in your database, and now you have all the information you ever wanted on GoogleBot Previews’ capabilities. You have been able to do similar things in the past, but never with such visual flair!

Have fun!

Please note the awesome Angie didn’t edit this for me as it was vaguely time sensitive, so please forgive the grammar!