SIMPSON’S PARADOX EXPLAINED

Imagine a treatment with the following properties:

The treatment is good for men	(E1)
The treatment is good for women	(E2)
The treatment bad overall	(E3)

That’s the essence of Simpson’s paradox. But there is no such treatment. Statements (E1), (E2) and (E3) cannot all be true simultaneously.

Simpson’s paradox occurs when people equate three probabilistic statements (P1), (P2), (P3) described below, with the statements (E1), (E2), (E3) above. It turns out that (P1), (P2), (P3) can all be true. But, to repeat: (E1), (E2), (E3) cannot all be true.

The paradox is NOT that (P1), (P2), (P3) are all true. The paradox only occurs if you mistakenly equate (P1-P3) with (E1-E3).

1. Details

Throughout this post I’ll assume we have essentially an infinite sample size. The confusion about Simpson’s paradox is about population quantities so we needn’t focus on sampling error.

Assume that is binary. The key probability statements are:

	(P1)
	(P2)
	(P3)

Here, is the outcome ( means success, means failure), is treatment ( means treated, means not-treated) and is sex ( means male, means female).

It is easy to construct numerical examples where (P1), (P2) and (P3) are all true. The confusion arises if we equate the three probability statements (P1-P3) with the English sentences (E1-E3).

To summarize: it is possible for (P1), (P2), (P3) to all be true. It is NOT possible for (E1), (E2), (E3) to all be true. The error is in equating (P1-P3) with (E1-E3).

To capture the English statements above, we need causal language, either counterfactuals or causal directed graphs. Either will do. I’ll use counterfactuals. (For an equivalent explanation using causal graphs, see Pearl 2000). Thus, we introduce where is your outcome if treated and is your outcome if not treated. We observe

In other words, if we observe and if we observe . We never observe both and on any person. The correct translation of (E1), (E2) and (E3) is:

	(C1)
	(C2)
	(C3)

These three statements cannot simultaneously be true. Indeed, if the first two statements hold then

Thus, (C1)+(C2) implies (not C3). If the treatment is good for mean and good for women then of course it is good overall.

To summarize, in general we have

and, moreover (E3) cannot hold if both (E1) and (E2) hold.

The key is that, in general,

and

In other words, correlation (left hand side) is not equal to causation (right hand side).

Now, if treatment is randomly assigned, then is independent of and

and so we will not observe the reversal, even for the correlation statements. That is, when is randomly assigned, (P1-P3) cannot all hold.

In the non-randomized case, we can only recover the causal effect by conditioning on all possible confounding variables . (Recall a confounding variable is a variable that affects both and .) This is because is independent of conditional on (that’s what it means to control for confounders) and we have

and similarly, and so

which reduces the causal effect into a formula involving only observables. This is usually called the adjusted treatment effect. Now, if it should happen that there is only one confounding variable and it happens to be our variable then

In this case we get the correct causal conclusion by conditioning on . That’s why people usually call the conditional answer correct and the unconditional statement misleading. But this is only true if is a confounding variable and, in fact, is the only confounding variable.

2. What’s the Right Answer?

Some texts make it seem as if the conditional answers (P2) and (P3) are correct and (P1) is wrong. This is not necessarily true. There are several possibilities:

1. is a confounder and is the only confounder. Then (P3) is misleading and (P1) and (P2) are correct causal statements.

2. There is no confounder. Moreover, conditioning on causes confounding. Yes, contrary to popular belief, conditioning on a non-confounder can sometimes cause confounding. (I discuss this more below.) In this case, (P3) is correct and (P1) and (P2) are misleading.

3. is a confounder but there are other unobserved confounders. In this case, none of (P1), (P2) or (P3) are causally meaningful.

Without causal language— counterfactuals or causal graphs— it is impossible to describe Simpson’s paradox correctly. For example, Lindley and Novick (1981) tried to explain Simpson’s paradox using exchangeability. It doesn’t work. This is not meant to impugn Lindley or Novick— known for their important and influential work— but just to point out that you need the right language to correctly resolve a paradox. In this case, you need the language of causation.

3. Conditioning on Nonconfounders

I mentioned that conditioning on a non-confounder can actually create confounding. Pearl calls this -bias. (For those familar with causal graphs, this is bascially the fact that conditioning on a collider creates dependence.)

To elaborate, suppose I want to estimate the causal effect

If is a confounder (and is the only confounder) then we have the identity

that is, the causal effect is equal to the adjusted treatment effect. Let us write

to indicate that the formula for is a function of the distributions and .

But, if is not a confounder, does the equality still hold? To simplify the discussion assume that there are no other confounders. Either is a confounder or there are no confounders. What is the correct identity for ? Is it

or

The answer (under these assumptions) is this: if is not a confounder then the first identity is correct and if is a confounder then the second identity is correct. In the first case .

Now, when there are no confounders, the first identity is correct. But is the second actually incorrect or will it give the same answer as the first? The answer is: sometimes they gave the same answer but it is possible to construct situations where

(This is something that Judea Pearl has often pointed out.) In these cases, the correct formula for the causal effect is the first one and it does not involve conditioning on . Put simply, conditioning on a non-confounder can (in certain situations) actually cause confounding.

4. Continuous Version

A continuous version of Simpson’s paradox, sometimes called the ecological fallacy, looks like this:

Here we see that increasing doses of drug lead to poorer outcomes (left plot). But when we separate the data by sex () the drug shows better outcomes for higher doses for both males and females.

5. A Blog Argument Resolved?

We saw that in some cases is a function of but in other cases it is only a function of . This fact led to an interesting exchange between Andrew Gelman and Judea Pearl and, later, Pearl’s student Elias Bareinboim. See, for example, here and here and here.

As I recall (Warning! my memory could be wrong), Pearl and Bareinboim were arguing that in some cases, the correct formula for the causal effect was the first one above which does not involve conditioning on . Andrew was arguing that conditioning was a good thing to do. This led to a heated exchange.

But I think they were talking past each other. When they said that one should not condition, they meant that the formula for the causal effect does not involve the conditional distribution . Andrew was talking about conditioning as a tool in data analysis. They were each using the word conditioning but they were referring to two different things. At least, that’s how it appeared to me.

6. References

For numerical examples of Simpson’s paradox, see the Wikipedia article.

Lindley, Dennis V and Novick, Melvin R. (1981). The role of exchangeability in inference. The Annals of Statistics, 9, 45-58.

Pearl, J. (2000). Causality: models, reasoning and inference, {Cambridge Univ Press}.

Please comment on the article here: Normal Deviate

The post SIMPSON’S PARADOX EXPLAINED appeared first on All About Statistics.

I took a session yesterday on ‘Useful software for research’ for the new batch of research students. I hope sharing the slides may be useful to other research students. The software suggestions are prepared based on my experience and my friends experience. The slide is prepared in svg (jessyink template) and it is better viewed in firefox/chrome.

Let me know if you have any suggestions. I could incorporate those for future sessions.

The post Useful Softwares for Research appeared first on Fiddling with data and code.

Please comment on the article here: Fiddling with data and code

The post Useful Softwares for Research appeared first on All About Statistics.

I wanted to locate some uncluttered lounge space for one of the threads to emerge in comments from 6/14/13. Thanks to Stanley Young for permission to post this. 

 S. Stanley Young, PhD
Assistant Director for Bioinformatics
National Institute of Statistical Sciences
Research Triangle Park, NC

There is a relatively unknown problem with microarray experiments, in addition to the multiple testing problems. Samples should be randomized over important sources of variation; otherwise p-values may be flawed. Until relatively recently, the microarray samples were not sent through assay equipment in random order. Clinical trial statisticians at GSK insisted that the samples go through assay in random order. Rather amazingly the data became less messy and p-values became more orderly. The story is given here:

http://blog.goldenhelix.com/?p=322

Essentially all the microarray data pre-2010 is unreliable. For another example, Mass spec data was analyzed Petrocoin. The samples were not randomized that claims with very small p-values failed to replicate. See K.A. Baggerly et al., “Reproducibility of SELDI-TOF protein patterns in serum: comparing datasets from different experiments,” Bioinformatics, 20:777-85, 2004. So often the problem is not with p-value technology, but with the design and conduct of the study.

Please check other comments on microarrays from 6/14/13.

Please comment on the article here: Error Statistics Philosophy » Statistics

The post Stanley Young: better p-values through randomization in microarrays appeared first on All About Statistics.

ISOTYPE

Good infographics for statistical matters do not only need diagrams (like histograms or bar charts) but also lots of icons and symbols helping to illustrate the topic.

A pioneer in this field was Otto Neurath.

‘ISOTYPE – the International System Of TYpographicPicture Education – … was an early infographical form, originated in the 1930s by Austrian philosopher and curator Otto Neurath “as a symbolic way of representing quantitative information via easily interpretable icons”.’ (Information is beautiful)

More about transforming data into visualisations also in Marie Neuraths ‘The Transformer: Principles of Making Isotype Charts’ 

Today: ONS as an example of good visualisations for statistics

Statistical Agencies use visualisations in their daily information work. During the UNECE Work Session on the Communication of Statistics in Berlin (27-29 May 2013) Alan Smith OBE, Office for National Statistics (ONS,UK) gave a short overview and insight in this topic.

Two statements from his paper ‘Data Visualisation for the Citizen User: Making Better Graphics Quicker‘:

‘Data visualisation appeals to National Statistics Institutes (NSIs) because of its ability to engage users and increase the potential outreach of official statistics. But data visualisation is a broad field, with content types ranging from simple infographics through to sophisticated tools for exploratory data analysis. ……

More broadly, data visualisation offers NSIs an opportunity to exploit their expertise in formats which boost user engagement and readership. It also carries with it the highly desirable side effects of boosting relationships with the media and reputational benefits virtually everywhere else. A final note of caution, however, is that these visualisations should be centred on the expertise of the NSI, not based on a notion of style over content – others do that better.’

.

Please comment on the article here: Blog about Stats

The post Infographics and ISOTYPE and NSIs appeared first on All About Statistics.

Brett Keller points to a recent news article by Sharon Begley and Robin Respaut:

A lot of doctors, hospitals and other healthcare providers in the United States decline to treat people who can’t pay, or have inadequate insurance, among other reasons. What sets CTCA [Cancer Treatment Centers of America] apart is that rejecting certain patients and, even more, culling some of its patients from its survival data lets the company tout in ads and post on its website patient outcomes that look dramatically better than they would if the company treated all comers. These are the rosy survival numbers . . .

Details:

CTCA reports on its website that the percentage of its patients who are alive after six months, a year, 18 months and longer regularly tops national figures. For instance, 60 percent of its non-small-cell lung cancer patients are alive at six months, CTCA says, compared to 38 percent nationally. And 64 percent of its prostate cancer patients are alive at three years, versus 38 percent nationally.

Such claims are misleading, according to nine experts in cancer and medical statistics whom Reuters asked to review CTCA’s survival numbers and its statistical methodology.

The experts were unanimous that CTCA’s patients are different from the patients the company compares them to, in a way that skews their survival data. It has relatively few elderly patients, even though cancer is a disease of the aged. It has almost none who are uninsured or covered by Medicaid – patients who tend to die sooner if they develop cancer and who are comparatively numerous in national statistics. . . . Accepting only selected patients and calculating survival outcomes from only some of them “is a huge bias and gives an enormous advantage to CTCA,” said biostatistician Donald Berry . . .

What I really like about this article is how it combines quantitative information with qualitative interviews:

Carolyn Holmes, a former CTCA oncology information specialist in Tulsa, Oklahoma, said she and others routinely tried to turn away people who “were the wrong demographic” because they were less likely to have an insurance policy that CTCA preferred. Holmes said she would try to “let those people down easy.” . . .

The ads also challenge viewers to “compare our treatment results to national averages.” Doing so, on the company’s website, shows that CTCA’s reported survival outcomes regularly beat those averages.

Experts in medical data who reviewed CTCA’s claims for Reuters say those claims are suspect because of what they called deviations from best practices in statistics – in particular, comparing its carefully selected patients to those nationwide.

“It makes their data look better than it is,” said Robert Strawderman, professor and chairman of biostatistics at the University of Rochester. “So the comparisons used to suggest that CTCA has better survival rates are pretty meaningless.”

This is horrible! Of course, we do something similar at Columbia—we accept only the very best students—but that’s a bit different, I think, in that we are providing an educational experience that is designed to work with the most-prepared students. In contrast, I doubt that there’s any particular reason for CTCA to restrict its cancer treatments to the least-sick patients.

The post “Behind a cancer-treatment firm’s rosy survival claims” appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post “Behind a cancer-treatment firm’s rosy survival claims” appeared first on All About Statistics.

Please comment on the article here: Junk Charts

The post Hard work pays off appeared first on All About Statistics.

I am not a big fan of the macro language, and I try to avoid it when I write SAS/IML programs. I find that the programs with many macros are hard to read and debug. Furthermore, the SAS/IML language supports loops and indexing, so many macro constructs can be replaced by standard SAS/IML syntax.

Nevertheless, many SAS customers use macro constructs as part of their daily SAS programming tasks, and that practice often continues when they write SAS/IML programmers. A customer recently asked a question about the macro language that required knowledge of the way that macro variables are handled within a SAS/IML loop. This post shares my response.

Here's the crux of the customer's question. Run the following SAS/IML program and see if you can understand why it behaves as it does:

proc iml;
i = 7;
call symputx("j", i);    /* 1. Put value of i into macro variable j */
y1 = &j;                 /* 2. Assign y1 the value of &j            */
print y1;                /* success! */
 
y = j(1,4,.);
do i = 1 to ncol(y);     /* 3. Start processing the DO block of statements */
   call symputx("j", i); /* 4. Put value of i into macro variable j */
   y[i] = &j;            /* 5. Hmmmm, what does this do inside the loop? */
end;
print y;                 /* Not what you might expect? */

As you can see from the output, the first use of the macro variable (outside the DO loop), works as expected. But the second does not. The customer wanted to know why the elements of y are not set to 1, 2, 3, 4 within the loop.

The key point to remember about macro variables is that SAS code never sees them. Macro variables are evaluated by the macro preprocessor at parse time, not at run time. The SAS/IML code never sees &j, only the constant value that the preprocessor substitutes for &j.

It is also important to remember that PROC IML is an interactive procedure. (The "I" in IML stands for interactive!) Each statement or block of statements is parsed as it is encountered, as opposed to the DATA step, which parses the entire program before beginning execution.

Let's examine the program step-by-step to understand why the first construct works but the second does not. The following steps refer to the numbers in the program comments:

1. The value of the SAS/IML scalar i is copied (as text) into the macro variable j.
2. The statement is encountered. The value of the macro variable j is substituted by the macro preprocesser. Then the statement is executed. The SAS/IML variable y1 is assigned to the value 7.
3. A DO loop is encountered by the SAS/IML parser. The parser finds the matching END statement and proceeds to parse the entire body of the loop in order to check for syntax errors. This parsing phase occurs exactly one time. Because the block of statements contain a macro variable, the macro preprocessor substitutes the value of the macro variable j, which is 7.
4. For each iteration, the value of the SAS/IML scalar i is copied (as text) into the macro variable j.
5. For each iteration, the ith element of the y vector is assigned the value 7. In particular, this statement does not contain a reference to the macro varible j.

To the casual reader of the program, it looks like &j will have a different value during each step of the iteration. But but it doesn't. The expression &j is resolved at parse time. SAS/IML parses the entire body of the DO loop once, before any execution occurs, and at parse time the expression &j is 7.

There is a way to get what the customer wants. The SYMGET function retrieves the value of a macro variable at run time. Therefore the following statements fill the vector y with the values 1 through 4:

do i = 1 to ncol(y);
   call symputx("j", i);
   y[i] = num(symget("j"));  /* get macro value at run time */
end;
print y;                     /* Yes! This is what we want! */

For me, this blog post emphasizes three facts:

• Always remember that macro substitution is done by a preprocessor, which operates at parse time.
• The SAS/IML language parses an entire block of statements (between the DO and END statements) one time before executing the block.
• Mixing macro code and SAS/IML statements can be confusing and hard to debug. When you have the option, use SAS/IML language features instead of relying on macro language constructs.

Please comment on the article here: The DO Loop

The post Macros and loops in the SAS/IML language appeared first on All About Statistics.

Yet another puzzle which first part does not require R programming, even though it is a programming question in essence:

Given five real numbers x₁,…,x₅, what is the minimal number of pairwise comparisons needed to rank them? Given 33 real numbers, what is the minimal number of pairwise comparisons required to find the three largest ones?

I do not see a way out of considering the first question as the quickest possible sorting of a real sample. Using either quicksort or heapsort, I achieve sorting the 5 numbers in exactly 6 comparisons for any order of the initial sample. (Now, there may be an even faster way based on comparing partial sums first… I just do not see how!) Update: Oops! I realised I made my reasoning based on a reasonable case, the correct answer is indeed 7!!!

For the second part, let us start from the remark that 32 comparisons are needed to find the largest number, then at most 31 for the second largest, and at most 30 for the third largest (since we can take advantage of the partial ordering resulting from the determination of the largest number). This is poor. If I instead use a heap algorithm, I need O(n log{n}) comparisons to build this binary tree whose parents are always larger than their siblings, as in the above example. (I can produce a sort of heap structure, although non-binary, in an average 16×2.5=40 steps. And a maximum 16×3=48 steps.) The resulting tree provides the largest number (100 in the above example) and at least the second largest number (36 in the above). To get the third largest number, I first need a comparison between the one-before-last terms of the heap (19 vs. 36 in the above), and one or two extra comparisons (25 vs. 19 and maybe 25 vs. 1 in the above). (This would induce an average 1.5 extra comparison and a maximum 2 extra comparisons, resulting in a total of 41.5 average and 49.5 maximum comparisons with my sub-optimal heap construction.)  Once again, using comparisons of sums may help in speeding up the process, for instance comparing numbers by groups of 3, but I did not pursue this solution…

If instead I try to adapt quicksort to this problem, I can have a dynamic pivot that always keep at most two terms above it, providing the three numbers as a finale result. Here is an R code to check its performances:

quick3=function(x){

comp=0
i=1
lower=upper=NULL
pivot=x[1]

for (i in 2:length(x)){

if (x[i]<pivot){ lower=c(lower,x[i])
}else{

upper=c(upper,x[i])
if (length(upper)>1) comp=comp+1}

comp=comp+1

if (length(upper)==3){

pivot=min(upper)
upper=sort(upper)[-1]
}}

if (length(upper)<3) upper=c(pivot,upper)

list(top=upper,cost=comp)
}

When running this R code on 10⁴ random sequences of 33 terms, I obtained the following statistics, I obtained the following statistics on the computing costs

</p>
<p style="text-align: justify;">> summary(costs)
Min.   1st Qu.  Median  Mean    3rd Qu.  Max.
32.00   36.00   38.00   37.89   40.00   49.00

and the associated histogram represented above. Interestingly, the minimum is the number of comparisons needed to produce the maximum!

Reading the solution in Le Monde in the train to London and Bayes 250, I discovered that the author suggests a 7 comparison solution in the first case [compare A and B, C and D = 2 comparisons; if A>B and C>D, compare A and C and get, say A>C>D = 1 comparison; insert E in this ordering by comparing with C and then A or D = 2 comparisons, obtaining e.g. A>E>C>D; conclude by inserting B by first comparing with C then with D or E = 2 comparisons] and a 41 comparison solution in the second case. He is (or I am!) I was clearly mistaken in the first case while the quick3 algorithm does 41 or less most of the time (90%)  but not always.

Please comment on the article here: Xi'an's Og » R

The post Le Monde puzzle [#825] appeared first on All About Statistics.

Intro

I recently had to create some sites quickly. After evaluating a few options, setting up a wordpress multisite seemed like a good option.

In order to make this change, I setup a wordpress multisite installation with domain mapping. A multisite installation is when one wordpress install lets you run multiple websites. I like multisite because it enables me to flexibly manage multiple websites with less duplication of effort than a single wordpress installation for each website would allow me.

Wordpress multisite normally works with subdomains (ie mail.google.com), but I combined the multisite mode with domain mapping to enable top-level domains to be used for each sub-site (so, we have vikparuchuri.com).

I am not a sysadmin by trade (what am I by trade, anyways?), but some new tools make it really simple to build repeatable configurations. I can’t stress the repeatable part enough. If you setup an installation “by hand” and run a lot of manual system commands, it will be extremely hard to reproduce if you need to run another site, or if you want to backup and re-initialize your site with different hardware. It may take more initial work to make something repeatable, but it is well worth doing.

Feel free to contact me if you have any questions about this process. Note that these steps have been tested only on Ubuntu 12.10. It will most likely work with windows, but some steps may need to be modified.

High level questions

What will this post help me do?

• Setup your own wordpress server for your personal blog.
• Setup your own wordpress server for several of your own websites.
• Setup a wordpress server that allows users to register their own sites automatically.

Why setup your own wordpress server?

• Often cheaper than using a hosted solution.
• You get more control over your data.
• Can install plugins/themes as you want.
• Quicker to make system-level changes such as package installation, switching domains, etc.
• Easier and more customizable scaling.

Why use multisite?

These instructions can be used to setup a single site or a multisite, but there are some advantages to using a multisite rather than setting up several single sites.

• Higher upfront time to setup, but much easier to setup each individual site after that.
• Central plugin and backup management.
• Easier to scale and manage.

Technologies Used

To enable the configuration steps to be repeatable, I used the following technologies:

• EC2 is an Amazon service that lets you create or shutdown servers on an on demand basis. It is great for deploying websites in seconds.
• Route 53 Route 53 is Amazon’s DNS service. It makes it incredibly simple to setup DNS records.
• CloudFormation is another Amazon service that lets you use templates to create resource “stacks”. These resources can be servers (EC2), databases (RDS), and so on. This lets you easily create and manage configurable resources. In our case, it allows us to create our server and associated resources very easily.
• Ansible is an open source project that allows for idempotent commands to be run by sshing into a server or group of servers. In our case, it allows us to very easily configure our wordpress server.

Getting the code

To get started, we first need to grab the wordpress-deployment repository:

1

git clone git@github.com:VikParuchuri/wp-deployment.git

Git is a version control system, and github is a social coding tool. If you haven’t used git before, github has some good tutorials.

The git clone command will make a new directory where you cloned it called wp-deployment.

Starting the cloudformation stack

After you have cloned the repository, you will be able to find the cloudformation template at wp-deployment/cloudformation/wordpress.json .

We will now need to login to an existing AWS account to use this template. See Amazon AWS for details on making an account.

One you login, you should be at the management console. The management console allows you to interact with AWS resources. In this case, we care about the cloudformation section.

Once you are in the cloudformation console, you will be able to click on “create stack”. Note that creating a stack will cost money. You can use this tool to estimate your cost.

After clicking on “create stack”, you will need to choose to upload the wordpress.json template.

Once you fill in the stack name and click “next”, you will come to the “specify parameters” screen. This is where the extensibility of cloudformation comes to the fore. This template lets you specify a few different variables.

• SSHLocation – If you want to restrict SSH access, set this. Otherwise, leave it at the default.
• EnvironmentTag – If you want a stage or sandbox instance to test with, change this. Otherwise, leave at prod. This will not change anything with the instance, it will only change how the instance is tagged. Ansible finds instances by how they are tagged (more on this later), so this is only if you want multiple types of servers with different configurations.
• ApplicationTag – Again, this is only a tag, so it only affects how the instance is discovered. I would recommend leaving it as the default.
• KeyName – In your EC2 setup, you will need to specify keypairs that can SSH into your instances. See the user guide for more info on this. The KeyName is the name of the keypair that you want to use. You will need to have it registered with EC2, and the private key will need to be on your computer.
• InstanceType – How large of an EC2 instance you want to create. I recommend not using the t1.micro, as it has not been fully tested (but if you are adventurous, feel free to test!).

Once you have set these, you will need to check the box that says “I acknowledge that this template may created IAM resources.” IAM resources are an Amazon feature that allow for multiple user roles with various access permissions. In this case, we are creating a user for our wordpress server that has limited access.

You can now hit “continue” through the next two screens (add tags and review). Amazon will now get to work creating your stack!

What is in the stack?

While the stack is being created, let’s talk about what is actually being made.

IAM User

As I alluded to before, cloudformation is making an IAM user. IAM gives us greater security, because we are not keeping our main AWS credentials on the server we are making. In this case, we are making a user that can access Amazon S3 (file storage), and send email via Amazon SES. Access to S3 allows the instance to be “bootstrapped” with some needed applications when it is created. Basically, the application downloads some basic applications from S3 and does some initial configuration. This makes it simpler for us down the line. Access to SES will let our wordpress instance send email to users (if we enable it), and it will let us backup our wordpress installation to S3 (again, if we enable it).

EC2 Server

The template will also create a server. This server will run wordpress, once we set everything up. It will need to have access to a database (or run a database locally). It will run the Ubuntu OS.

ELB

An ELB is an elastic load balancer. It basically redirects from an external facing URL to a group of servers. It is generally intended to balance load between several machines. In our case, it serves two purposes. Amazon Route 53 (which we will go into later) does not allow naked domain redirection (ie vikparuchuri.com instead of www.vikparuchuri.com) unless we use an ELB. The ELB also allows us to swap servers in/out on the backend however we want later.

Security Group

A security group determines who can access our server, and from what ports.

The database

You may have noticed that I did not include a database in the previous description of the stack. This is because I wanted to make it as flexible as possible what database you use. Also, including the database in a stack is a bit fraught with danger, because you risk losing all of your data if you accidentally delete a stack.

You can either create a database locally on the EC2 server you have just made, or make a separate database server using Amazon RDS. In my case, I used Amazon RDS because it makes it more flexible to add/remove EC2 servers without losing data.

Feel free to use either option. For the purposes of this tutorial, I will go with a local database, because RDS is a bit more complex to setup.

Stack outputs

The stack should now have finished coming up. If you see a red light and the status “ROLLBACK_COMPLETE”, you should check “Events” under the stack detail to find the reason for the rollback. Often, it is something like having typed a configuration setting in incorrectly.

Now that the stack has been created, you can click on the stack in the console and check on the outputs.

The outputs will tell you the ELB address (referred to from hereon as ELBAddress) and the EC2 server address (referred to from hereon as ServerAddress). Save both of these somewhere: you can access them here anytime, but we will need them in several places.

Setting up a local database

We can now setup a local database for the wordpress instance. Feel free to skip this if you will be using an external database. Regardless of what database server you are using, it is still nice to have a user just for wordpress with limited access, though (this will automatically be setup later on).

Install mysql server

Let’s ssh into our instance:

1

ssh ubuntu@ServerAddress

We will need to install mysql:

1

sudo apt-get install mysql-server

Set whatever root password you want, but make sure you save it somewhere.

Setup database/user/permissions

The following steps in this section (creating a database, a user, and granting the relevant permissions), will be done automatically. Only do it yourself if you want to understand the process or debug.

Now, let’s create a database, a database user, and give the user the right permissions.

1

mysql -u root -p

Type in the password you set during the installation of mysql-server when it asks for a password.

Once you are in the mysql shell:

1
2
3

CREATE DATABASE wordpress;
CREATE USER wordpress IDENTIFIED BY '[insert password here]';
GRANT SELECT,INSERT,UPDATE,DELETE ON wordpress.* TO 'wordpress'@'localhost';

Save your wordpress user password somewhere.

We are now done setting up the database, and we can move on to the other steps. Use exit to terminate the ssh connection.

Setup Route53 to forward our domain to the ELB

Route53 is Amazon’s DNS management system, as I mentioned earlier. It will allow us to point users who go to www.oursite.com to our ELB. Feel free to skip this if you don’t need this, or if you plan to manage your DNS some other way. You will always be able to access your site via ELBAddress.

To use Route53, first go to the Route53 Console.

Then, click on “Create Hosted Zone”.

This will pop up a box on the right that lets you enter your domain name. Enter your domain name without the www, and then click create hosted zone at the bottom.

Your domain will now appear in the center panel. Select your domain, and then look at “delegation sets” at the right.

These are your namesevers, and you will need to set these as the DNS records with your registrar. The AWS help at the top right will give you information on this if you have questions.

Once you have setup the nameserver configuration, you will need to setup the records to point at your wordpress elb (which will point to your server).

To do this, you will need to setup two alias records, one for the “naked” domain (vikparuchuri.com), and one for the full domain (www.vikparuchuri.com).

Select your domain in the Route 53 control panel, and the click on “go to record sets” at the top left. This will bring you to a detail view of the record set.

Now, you can click on “Create record set”, which will pop up a box at the right.

We will be making an alias record that points at our ELBAddress.

Click create record set when you are done, and then do the same for the full domain.

We are now setup as far as what we need for the wordpress install.

Setup for wordpress deployment

We are now ready to do our basic wordpress configuration with ansible. This will setup a basic single-site wordpress install.

Setup ansible

On our local machine, let’s go to the wp-deployment folder that we created earlier with git clone:

1

cd wp-deployment

You will need to be running python 2.6+ before doing the following step. See this link for how to install python on ubuntu. You will also need pip, which you can get by doing easy_install pip after python is installed. See this link for a description of how to do this on a mac, and this link for how to do this on windows.

Now, using the python package manager pip, let’s install the requirements. Feel free to use a virtualenv or not use one for this. A virtualenv is a python tool that allows you to isolate environments for each of your applications. I highly recommend looking at virtualenvwrapper if you choose to use a virtualenv.

1

pip install -r requirements.txt

Set secret variables

Ansible has a concept called playbooks. Each playbook will match a certain subset of the potential hosts (in this case, the hosts are our ec2 instances, and subsets are the ones with the correct tags, which is why it was important to set tags earlier).

Playbooks will run certain commands on a certain set of hosts. In this case, we want to run the commands to correctly setup wordpress.

Get the template file from secrets/vars/wordpress_prod_vars.yml.template. Then, edit the values to reflect what you need.

The values from auth_key to nonce_salt are wordpress internal secret variables (salts). You can generate random ones here. When you insert them into the template, make sure that you put them between the quotes. The single quotes need to be there.

Your final template should look something like this (don’t use these salts, generate your own!).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

database_user: wordpress
database_password: INSERT_PASSWORD_HERE

database_host: localhost
database_root_user: root
database_root_password: INSERT_ROOT_PASSWORD_HERE

auth_key: '!UCVyMfA4q6~GOt]hr9!{9H/Ec8*rs9,9Ow|~n0pEbRacmLnD~Bb]GC9DW9/n+;k'
secure_auth_key: ':rg-V/}0h`0]Sx]/tR6YByYwulzT<TXd_tD^&CBY$+b$H`yxA{*`Bv! -Mmwqp ('
logged_in_key: '/QA}nvBwLjVIY>CPH(.}FQ%&)@e((yCn_V`RrVg>@[YiyX]DX9{q@&@H|!O7PiE@'
nonce_key: 'O!,(tiA;C,$]v&:,N6^60wT<-hq/p sF])0)Y9(pQ`u0!.J,;KYU]n%oTD]$o{Oq'
auth_salt: '>@E7siI)e|;rc@ qwo^9GX}D O+9DEh6@hd%PifC/yyvaH?c8)+7swV-D9=%WJF1'
secure_auth_salt: 'YLKK+W&Lrx-/4@r<1[AW9>v&Sg|HnZ. c)N`NNvBe!gc=`5[bqjSisslF+:L1x G'
logged_in_salt: '@AzQQr#}5;QD<iDSu#)|wM(UQ7?LV#I|F` ]=:LnFN`-!)1i!!I>a96iaqE*y[1+'
nonce_salt: '39OSa0v~!=vh0j[YXOmR?,tq-G]x:uphoNaO)Hj..&|2Dg@G:S|#}QZ@49+b46j5'

elb_address: INSERT_ELBAddress_HERE

A note on the elb_address. This is the site where your server will be accessed from. If you have a domain that you want to access your site from that you just setup with route53 (ie vikparuchuri.com), then use that. For multisite, make it the primary address you will be accessing from. If you don’t have a domain, using the ELBAddress is fine.

Once your template looks good, you can save it without the .template extension (wordpress_prod_vars.yml).

Boto configuration

Ansible needs to find your server, which it does via boto, which is a python utility to connect to Amazon AWS. To configure boto, you will need to make a .boto file with your AWS credentials.

Please see these instructions for more information on configuring boto.

If boto is setup correctly, you will be able to run:

1
2

cd wp-deployment/playbooks
./ec2.py

This should show you all your running EC2 servers.

Deploying wordpress via ansible

These instructions will setup a single site via wordpress, which can then be extended to be a multisite.

Inspecting the playbooks

We will need to first go to the playbooks directory.

1

cd wp-deployment/playbooks

If you look at the folder, you will see that there are .yml files, which are the playbooks, and a roles folder, which contains the code that the playbooks actually run on the hosts.

If you look at the roles folder, you will see two roles: “mu” and “wp”. The wp role will make a wordpress installation on a site, and the mu role will convert it to be a multisite installation.

Both roles have associated variables in their vars folder that you can edit if you want, but there is no real reason not to use the defaults.

Running the playbook

We can now run ansible-playbook -vvv --user=ubuntu wp_prod.yml -i ./ec2.py -c ssh, and it will connect to our EC2 server and configure it with wordpress.

If the command creates an error, you may have done the cloudformation configuration incorrectly and tagged your machines improperly. You can look at the tags and fix this in the “hosts” section of the wp_prod.yml playbook.

Once the playbook finishes running, you can go to YOUR_SERVER_NAME/wp-admin/install.php on your server to begin wordpress installation. YOUR_SERVER_NAME should be what you entered for elb_address in the template. In this example, I set up route 53 to point to my ELB, and used the route 53 domain.

Setting up wordpress

Once you can get to YOUR_SERVER_NAME/wp-admin/install.php, you can setup your site through the interface. One note of caution : do not use admin as your username. There are people who scan for the admin username and try to guess the password through brute force attacks.

Once you finish the setup screens, congratulations! You have setup a single user wordpress site.

Activating multisite mode

Activation

Now, to activate multisite mode for your wordpress install, you will have to go to the wp-admin, and then click on tools/network setup. You can also go to the url YOUR_SERVER_NAME/wp-admin/network.php.

You can then type in some settings to setup network mode. Make sure you select sub-domains!

Post activation

After you hit the install button, you will come to a screen that asks you to do some configuration. You can ignore those steps for now, as we will be doing that through an ansible playbook.

If you see a “Warning! Wildcard DNS may not be configured correctly!” in red at the top, this means that you have not setup *.YOUR_SERVER_ADDRESS.com to redirect to the ELB. You can do this using Route 53 alias records (see the previous section on this), or you can skip it for now. All this means is that whenever you make another site in the multisite network, you will need to setup a redirect from MULTISITE_NAME.YOUR_SERVER_ADDRESS.com to the ELB (you need to do this even if you are using separate domain names for each of your sites on multisite).

Configuring multisite mode

After activating multisite mode, you can then run the mu playbook to setup multisite.

1
2

cd wp-deployment/playbooks
ansible-playbook -vvv --user=ubuntu  mu_prod.yml -i ./ec2.py  -c ssh

And now you will be able to go to wp-admin on your site, and login. Multisite mode is now active, and you will be able to make a network of sites with their own domain names.

Making new sites

Set up a new site with a subdomain

To make a new site, go to YOUR_SERVER_ADDRESS/wp-admin/network/site-new.php, and make a new site with a subdomain.

Set up the DNS for that subdomain to redirect to your ELB, even if you are not using Route 53. If you already have a wildcard redirect, then you can skip this.

Using a top level domain

If you want to use a top level domain, such as test.com, for the site:

Go to domain mapping in YOUR_SERVER_ADDRESS/wp-admin/network/settings.php?page=dm_domains_admin and set the id of the site (you can get this from the url when you click on a site in sites), and the domain. Make sure that primary is checked.

You will have to redirect the domain (www.test.com) and the apex domain (test.com) to your ELB. You can do this via Route 53, and the instructions are above.

Modifications/Contributions

The code for the cloudformation template and the ansible playbooks is on github, so please feel free to fork and submit a pull request if you want to change something.

Future post topics

There are some other useful things that I did to make wordpress easier to manage, and these include: * Setting up email via Amazon SES * Automated backups via xcloner * Migrating from blogger to wordpress and setting up redirects * Setting up a database using Amazon RDS * Setting up email/calendar using Google Apps

I intend to post about these down the line, as time permits. Please let me know if any of them are particularly interesting.

Please comment on the article here: Vik's Blog

The post Creating a wordpress single or multisite install using cloudformation and ansible appeared first on All About Statistics.

re:log [opendatacity.de] by German data designers OpenDataCity reveals the movements of about 6,700 different electronic devices during re:publica 2013, a prestigious European conference on the topic of Digital Society.

A dynamic map of the conference location shows the approximate locations of the devices when they were connected to the local WiFi hotspots. An interactive timeline underneath allows to explore the dynamic changes over time, while a rectangular area can be drawn to more specifically highlight and follow a smaller amount of dots.

The visualization was based on tracking the MAC addresses of the devices according to the WiFi hotspot they were connected to. This data, which can be downloaded, was fully anonymized, yet the authors mention their desire to allow people to look up their own MAC address in the future.

Please comment on the article here: information aesthetics

The post re:log: Tracking the Movements of Conference Attendees via WiFi appeared first on All About Statistics.