Diego Valle's Blog

Maps of Mexico City

2013-04-09T01:30:00.000-05:00

Just some maps of the Mexico City metro area based on the 2010 census. You can search by address and switch to satellite view. If the maps seem slow I suggest you use chrome.

To calculate the margination index I used the first principal component of the percentage of the population with at least one year of college, households with a dirt floor, households without electricity, etc

When it comes to attracting young women to their neighborhoods the mirrreys beat the hipsters hands down (though the suburbs are probably not the best place to actually meet women).

I also wonder what's up with all the non religious persons in La Condesa and Roma Norte. The UNAM I can understand, I'm guessing I didn't get the memo on the hipster heresy.

There's also a negative relationship between car ownership and population density, though obviously a lot more variables are involved. The rich areas and the outer ring surrounding the urban center are low in population density, while the poor mostly live in high population density areas.

GitHub

Visualising Change in Presidential Votes

2013-03-05T07:33:00.000-06:00

Click on the image to visit the interactive version (Spanish)

I liked this visualization by the New York Times of the US presidential election so much I decided to recreate it using Mexican data, and in the process added a slope graph. So go visit my new visualization of the change in presidential vote from 2006 to 2012 at the municipality level. I recommend using chrome to view the visualisation, there's also a Spanish version.

The PRI gained votes in almost all the big municipalities save for Villahermosa (Centro). The home state of AMLO.

Not everyone was happy with the post-election protests of the PRD candidate. Interestingly, he did gain votes in several of the municipalities that make up the Monterrey and Guadalajara metro areas.

People in Villahermosa must really hate the PRI. I was not surprised to see the PAN lose votes in Culiacán, Juárez, Chihuahua, Cuernavaca, Tijuana, Zapopan, Guadalajara, Monterrey...

Scatter plots of votes in 2006 vs votes in 2012:

P.S. The code for the interactive chart is here

Download shapefiles of Mexico

2013-02-27T08:01:00.000-06:00

I've created a set of scripts to download and recode shapefiles from the IFE and INEGI. Once you run the scripts you'll find in the map-out directory:

distritos: Shapefile of the electoral distritos (districts)
secciones-inegi: Shapefile of electoral secciones (precincts) with both the ife and inegi codes for the municipalities each seccion belongs to
estados: Shapefile of the Mexican states according to the INEGI
localidades: Shapefiles of the rural localities and the polygons of the urban ones
municipios: Shapefile of the municipalities of Mexico according to the INEGI
rdata-secciones: serialized secciones (precincts) map as an R object

Since the IFE uses a different coding standard for the municipalities of Mexico than the INEGI, I've recoded the municipality codes so that they match the ones the INEGI uses.

Ecatepec, México according to the INEGI is 15 033, while according to the IFE it's 15 034
Guadalajara, Jalisco according to the INEGI is 14 039, while according to the IFE it's 14 041

These codes are only available for the secciones electorales (precincts) shapefile and they are contained in the variables:

MUN_INEGI: The inegi municipio codes
MUN_IFE: The original ife municipio codes that came with the file

The codebook for the the census data that comes with the distrito and sección shapefiles is in the FD_SECC_IFE.pdf file and the ife and inegi codes are in the ife.to.inegi.csv file

I also suggest you check out Rob Hidalgo's repository of public information (from freedom of information requests) which includes a shapefile of the municipios of Mexico according to the IFE (mine comes from the INEGI and they differ by a few municipalities)

P.S. Here's the code

Mexican Presidential Election by District

2013-02-11T21:18:00.000-06:00

Check out the interactive map I made of the results of the presidential election in Mexico. I even used hierarchical clustering to group Mexico into five clusters according to how people vote (obviously the clusters are very rough and subject to statistical error).

These are the median values for the percentage of the vote for each party in the clusters:

Interpretation	Cluster	PRD	PAN	PRI
PRI	1	0.27	0.20	0.46
PRI-PAN	2	0.23	0.34	0.38
PRI-PRD	3	0.40	0.16	0.37
PRD	4	0.54	0.15	0.27
PAN	5	0.22	0.48	0.26

Interactive map of the drug war in Mexico - 2011

2013-01-22T21:14:00.000-06:00

I've updated the interactive map I made last year to include the 2011 homicide data the INEGI recently released. Because of bad publicity the government will no longer update the drug war-related homicide database, so that dataset will forever remain stuck in September 2011.

The cool thing about the new version of map is that you can statistically correct some of the errors in the way homicides are recorded in Mexico.

Violent deaths can be classified as suicides, accidents, homicides, legal interventions and those of unspecified intent. There has been an upsurge in deaths of unspecified intent by firearm in Mexico at the same time all homicides increased, so it's natural to assume that some of those unspecified deaths were really homicides.

To correct the mistakes here is what I did:

Classify all deaths of unknown intent into accidents, suicides and homicides based on the age, sex of the victim and injury mechanism by which the death occurred. The method I used is similar to the one I used in my Juarez post. For example, if someone told you to guess the intent of the death of a 70 year old woman that died in Merida (one of Mexico's safest cities) by motor vehicle, you'd probably guess it was an accident. If you had to guess the intent of the death of a young male in Ciudad Juárez that died by firearm, you'd probably think it was an homicide.
Recode all deaths by legal intervention as homicides (the 19 deaths from the shootout between the army and the narcos in General Treviño and Mier were recorded as homicides) since it seems legal intervention deaths are not always classified correctly.
Reconcile the problems in the Federal District and Sinaloa
Since firearm accidents went from 6 in 2006 to a 100 in 2007 and again to 6 in 2008 I reclassified all firearm accidents in Baja California in 2007.
I added the mass grave in Taxco and the one in San Fernando since they don't appear or are incomplete in the dataset from the INEGI.

Here are the results from in terms of sensitivity and specificity from the classification:

Even with the corrections I made there are still some errors left in the database, hopefully in the future I'll figure out how to fix them.

Visit the map on facebook: https://www.facebook.com/narcomap

P.S. You can download the source from GitHub
P.P.S You can also download the program to clean the mortality database
P.P.P.S If all you need is the data you can download it from the datasets section of my website

Mapa interactivo de la guerra contra el narcotráfico - 2011

2013-01-22T11:31:00.001-06:00

Bueno, pues he actualizado el mapa interactivo de la guerra contra el narcotráfico para incluir los datos de muertes por homicidio en el 2011 que acaba de publicar el INEGI. Al parecer el gobierno ya no va a actualizar los homicidios debido al crimen organizado (también llamados de presunta rivalidad delincuencial) pero el mapa los sigue mostrando.

Para contar los homicidios asumo que en aquellos donde no se especifico el día de ocurrencia, ocurrieron en el mismo mes que fueron registrados.

También ajuste los datos del INEGI en el 2011 por aproximadamente 3% debido a que por la fecha de corte del 31 de Diciembre para registrar los homicidios no da tiempo de registrar todos los homicidios que ocurrieron durante el 2011. Asumí que el subregistro de homicidios ocurrió de la misma manera que en el 2010.

Aunque Culiacán y Navolato no son oficialmente un área metropolina las consideré como tal (y actualicé las demás áreas metropolitanas al 2010).

Pero lo más importante del mapa es que ahora se puede hacer un ajuste estadístico por errores y casos no reportados

Las muertes violentas se pueden clasificar en suicidios, homicidios, accidentes, operaciones legales y aquellas donde no se especificó la intención. Las muertes de intención no especificada por el mecanismo de arma de fuego han ido aumentando al mismo tiempo que se disparó la violencia y es normal pensar muchas de estas muertes en realidad son homicidios.

Usando k-nearest neighbors clasifiqué cada una de estas muertes como homicidios, suicidios y accidentes en base al sexo, el mecanismo de defunción (arma de fuego, asfixia, transporte, etc) y la edad del difunto.

Para asegurarme de no meter la pata a la hora de clasificar las muertes calculé la sensibilidad y la especificidad de los resultados guardando una muestra del 20% de los casos que no fue usada para entrenar el algoritmo:

A las operaciones legales también las clasifiqué como homicidios

En septiembre del 2010 hubo una balacera en los municipios de General Treviño, NL y Mier, Tamps en la que murieron 19 personas a manos del ejercito. En la base de datos del INEGI aparecen como muertos por homicidio cuando en teoría deberían haber sido registrados como muertes por "Operaciones de Guerra / Intervención Legal". Por este motivo recodifiqué las pocas muertes por operaciones legales que hay en la base de datos (20-40 al año) como homicidios.

Los homicidios que fueron registrados como muertes accidentales en el Distrito Federal también los clasifiqué para que fueran bien contados.

También corregí la discrepancia entre los homicidios totales y homicidios del crimen organizado en Sinaloa

Y por último agregué la fosa con 55 cuerpos que fue encontrada en Taxco y añadí la otra fosa que fue encontrada en San Fernando en abril del 2011.

A pesar de estas correcciones quedan algunos errores en la base de datos, así que espero en un futuro tener todavía mejores datos:

Visita el mapa en facebook: https://www.facebook.com/narcomapa

P.D. El código del mapa está disponible en GitHub
P.P.D. El código para limpiar la base de datos de mortalidad también está disponible
P.P.P.D Si nada más te interesan los datos también están disponibles para su descarga

Map of homicides in Monterrey

2013-01-15T07:22:00.000-06:00

Click on the image to visit the map

This a somewhat cleaned up 2d density map of homicides in Monterrey as reported by El Norte during all of 2012. Do take note that the map is of homicide locations rather than counts and that I removed a couple of non-homicide events that were present in the original data.

Monterrey is surrounded by mountains and if you notice in the map the area around Topo Chico "bleeds" into the mountain, so it would probably be a good idea to use a more complicated model, taking into account the natural boundaries of Monterrey, to estimate the density of homicides.

In this presentation by Arturo Cervantes you can see that the areas with the most homicides kind of match the locations of motor vehicle accidents. The central part of Monterrey is actually less population dense than other areas, but apparently everybody travels there for work or play. And since everybody is there, that's where most homicides happen.

Data Source: Mapa del Crimen 2012 from El Norte

The most dangerous days of the year

2012-12-18T22:09:00.000-06:00

When looking at days of the year with the most homicides a pattern jumps out: violence is significantly higher during certain holidays in Mexico compared to the rest of the year. This is a well studied pattern in other countries.

Most violent dates 1998-2007:

Date	Homicides
2000-Jan-01	93
1998-Dec-25	92
2001-Jan-01	89
1999-Jan-01	88
2005-Jan-01	80
1999-Dec-25	78
1998-Mar-01	76
1998-Jun-14	74
1998-Oct-25	73
1998-Nov-29	73

Six of the ten most violent days from 1998 to 2007 where either the first day of the year or Christmas. To check whether holidays were really more violent than the rest of the year I plotted the total number of homicides for the period 1998-2003, and 2005 (so each day after February would fall exactly once each day of the week). I excluded the drug war years to make the pattern more obvious.

Sum of daily homicides from 1998-2003, and 2005 (ten most violent dates):

Total Homicides (Sum of All Dates)	Day Number	Holiday
558	Jan-01	New Year
388	Dec-25	Christmas
357	Sep-16	Independence Day
281	Nov-02	Day of the Dead
270	Feb-15	Day After Valentine's Day
270	Dec-12	Virgin of Guadalupe
269	Dec-24	Christmas Eve
263	May-10	Mother's Day
259	Dec-31	New Year's Eve
256	May-03	Related to Labor Day?

I find it quite interesting that the day of the dead would be one of the most violent days of the year. In countries where most homicides are solved, most homicide victims knew their attacker, so perhaps hanging out with family and friends during the holidays is not such a good idea? Does that stress people out?

Performing an analysis of variance and testing for multiple comparisons with a Tukey test, the list of holidays above (coded as a binary variable), controlling for month of year and day of week, were expected to have between 28 and 39 extra homicides with a point estimate of 33 homicides.

fit <- aov(total.deaths ~ as.factor(holiday) + as.factor(month(date)) + as.factor(wday), 
            data = homicides)
TukeyHSD(fit)

Results for all days from 1998 to 2007:

##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = total.deaths ~ as.factor(holiday) + as.factor(month(date)) + as.factor(wday), data = homicides)
## 
## $`as.factor(holiday)`
##             diff   lwr   upr p adj
## TRUE-FALSE 33.46 28.37 38.55     0
##
## $`as.factor(wday)`
##         diff      lwr       upr  p adj
## 2-1 -11.9148 -13.3360 -10.49355 0.0000
## 3-1 -16.9419 -18.3645 -15.51927 0.0000
## 4-1 -18.3969 -19.8188 -16.97494 0.0000
## 5-1 -17.6014 -19.0226 -16.18016 0.0000
## 6-1 -16.3855 -17.8068 -14.96427 0.0000
## 7-1 -11.1132 -12.5344  -9.69192 0.0000
## 3-2  -5.0271  -6.4490  -3.60516 0.0000
## 4-2  -6.4821  -7.9033  -5.06083 0.0000
## 5-2  -5.6866  -7.1072  -4.26605 0.0000
## 6-2  -4.4707  -5.8913  -3.05016 0.0000
## 7-2   0.8016  -0.6189   2.22219 0.6398
## 4-3  -1.4550  -2.8776  -0.03238 0.0411
## 5-3  -0.6595  -2.0815   0.76240 0.8188
## 6-3   0.5564  -0.8656   1.97829 0.9109
## 7-3   5.8287   4.4068   7.25064 0.0000
## 5-4   0.7955  -0.6258   2.21670 0.6489
## 6-4   2.0113   0.5901   3.43259 0.0006
## 7-4   7.2837   5.8625   8.70495 0.0000
## 6-5   1.2159  -0.2047   2.63645 0.1506
## 7-5   6.4882   5.0677   7.90880 0.0000
## 7-6   5.2724   3.8518   6.69292 0.0000

Sunday is 1

Mexico's Drug War: 63,000 extra deaths in 6 years

2012-12-11T21:58:00.000-06:00

Six years ago Felipe Calderón declared war on the drug cartels soon after taking office by sending troops to his home state of Michoacán. Six years later the war still rages on as Enrique Peña Nieto starts his term of office, and everyone is taking a look at the legacy of the man who started the war.

There are newspapers and organizations that keep tallies of drug war-related deaths, which in itself is very laudable, but drug trafficking existed long before Calderón came to office and it will exist long after he leaves. Plus, there is also the issue that the government stopped publishing data on drug war-related deaths after September 2011, and its numbers usually came out higher than the private estimates.

To judge the events of the sexenio I'll focus first on counting the total number of homicides as accurately as possible for a blog post, then setting a baseline and counting the number of excess deaths. To start, there's the official INEGI data and SNSP data which everyone can access online. I'll order the INEGI homicides by date registered (instead of date of occurrence) and add firearm accidents since they:

Have been increasing
As we saw in my last post sometimes they resemble homicides more than accidents.

Year	INEGI Homicides	Firearm Accidents	Unknown Intent	Imputed INEGI Homicides	SNSP Homicide Reports†
2,004	9,328	366	2,957	10,339	11,658‡
2,005	9,918	271	2,932	10,942	11,255‡
2,006	10,449	265	2,793	11,414	11,775‡
2,007	8,864	699	2,376	10,205	10,253
2,008	14,005	555	2,567	15,246	13,193
2,009	19,804	456	2,920	21,065	16,117
2,010	25,757	630	3,594	27,508	20,681
2,011	27,212		5,630		22,487

† A homicide report may refer to more than one victim
‡ The State of Mexico was registering accidental deaths as if they were homicides

Notice how deaths of unknown intent were on a downward trend up to 2008 when the trend reversed at the same time homicides surged. As I detailed in this post and in my post about classifying deaths of unknown intent in Ciudad Juárez they:

Statistically resemble homicides and
Explain some of the difference in under-reported homicide deaths when compared with drug war-related homicides.

To classify deaths of unknown intent I first recoded legal intervention / operations of war deaths as homicides, I then excluded deaths that occurred outside of Mexico. Because of the anomaly in the Federal District I recoded all accidental deaths during January and February of 2007 in this federate entity as unknown. In Sinaloa I recoded accidental deaths by firearm as unknown as I detailed in my previous post. And in Baja California I set all firearm accidents as deaths of unknown intent in 2007.

I still have lots of doubts about the vital statistics in Chiapas, but since it is not a state that has suffered the worst of the drug war I'll leave it alone. Also the mass graves found in Guerrero seem to be missing from the data (add another 100?), but I'll ignore it. And in Tamaulipas even including deaths of unknown intent under-counts drug war-related homicides, but again I'll ignore it (add about another 200-400? homicides to my estimate).

I used linear discriminant analysis and k-nearest neighbours classify each and every death of unknown intent in the database (plus some accidents for the states I previously mentioned) taking into account the injury mechanism, and the age and sex of the victim. To make sure I wasn't overfitting the data I calculated the sensitivity ans specificity for each state or group of states (some states are pretty small):

The accuracy refers to the overall accuracy of the classification of homicides, accidents and suicides. The sensitivity and specificity refer only to homicides

I could have used fancier algorithms to try and get better classification performance, but the results were pretty good and I wanted something fast. Also take into account that since I have to use the raw data from the mortality database I only have data available up to 2010.

Here's what the more accurate monthly homicide time series looks like:

I used an ARIMA(0,1,1)(1,1,0)[12] regression on the monthly SNSP police reports to predict the INEGI homicides + firearm accidents + the deaths of unknown intent imputed to be homicides from January 2012 to November 2012 since no vital statistics data is available after 2011.

Do note the increase in Tamaulipas

For the baseline estimate of the number of deaths I simply used the average monthly deaths from January 2004 to November 2006 (increasing at 1% per year), I then substracted or added the total number of imputed deaths from the baseline estimate (some homicides were surely prevented by the drug war in the beginning stages) to obtain the excess deaths. The total excess imputed homicides for the Calderón sexenio was of 63,000 (61,000-66,000 95% CI), I rounded the numbers to the nearest multiple of 1,000 since it doesn't make any sense to report the number with much accuracy.

There are a number of problems with my estimate:

I used data on police reports from the SNSP, but if I used drug war homicide data from Milenio I would have goten much higher estimates since deaths during the first eleven months of each year increased from 11,343 in 2011 to 11,412 in 2012.
The more or less arbitrary baseline is open to criticism. According to the Secretary of Interior Alejandro Poiré they found a country "infested by rats". Calderón supporters would put the baseline much much higher than the one I used, though I have the feeling this is just an ad-hoc justification since they never provided any evidence for this. Then there is also the fact that the government considers most of the dead to have been criminals, so we have the president (in)famously stating "If you see dust in the air, it's because we're cleaning house," supporters of the war would argue that those who lost their lives in a certain way deserved their fate.
Saying that Mexico was on the verge of a drug war before the president took office is not the only view since each generation of Mexicans was becoming less violent before the drug war (though with important period effects), and Mexico had had episodes of localized drug violence before in mid-ninties in Ciudad Juárez and at the turn of the century in Tijuana without plunging into a extended drug war, so maybe the baseline is set too high.

I'm still missing mass graves that haven't been found, and bodies that simply disappeared without leaving a trace. One could combine the various lists of disappeared kept by the PGR and SSP to produce an estimate of the number of missing persons in Mexico, but you would still have to match them somehow to the mortality datase since many o the dead are never fully identified. This means my estimate is a lower limit.

How a Mexican state ended up with more drug war homicides than total homicides

2012-12-05T21:59:00.000-06:00

During 2007 and 2008 the Mexican state of Sinaloa had more drug war-related homicides than total homicides. This should in theory be impossible since drug war homicides are a subset of total homicides. How did this happen?

Here is a chart from my old post highlighting the monthly difference between the vital statistics data and drug war-related homicides (Dec 2006- Dec 2009):

Just two weeks ago 8 bodies were found in Sinaloa, I can only image how difficult it must be to determine the exact dates the homicides occurred in some cases. So it's not a big deal if a month ends up having 5 or so extra drug war homicides than total homicides. But when drug homicides exceed total homicides for a year and a half as it did in Sinaloa, you know something is wrong.

Most homicides in Sinaloa are caused by firearms, and given the other errors in the mortality database it seems one should look if the discrepancy in the numbers lie in an excess of firearm deaths.

Here's a table of the yearly vital statistics data on homicides, firearm accidents and drug war-related homicides:


Year	All Homicides	Drug-Related Homicides	Accidental Firearm Deaths
2004	387	NA	61
2005	450	NA	47
2006	451	NA	26
2007	387	426	204
2008	869	1,084	198
2009	1,422	1,059	36
2010	2,277	1,815	67†

† Preliminary estimates undercounted by about 6%

I highlighted the years where drug homicides exceeded all homicides. As you can see the anomalous years also had an extremely high number of firearm accidents (I used the External Cause of Injury Mortality Matrix for ICD-10 to code accidents by firearm)

In fact, there's a very high correlation when looking at the unexplained drug homicides (the drug homicides left over when we substract the drug war-related homicides from the total homicides) and firearm accidents, but only during the anomalous years:

I don't think there can be any doubt that at least some of deaths by firearm accident correspond to those in the drug-war related homicide database during 2007 and 2008. Now the question is whether they are really homicides or accidents.

It seems natural that firearm accidents would rise along with homicides as the drug war ratcheted up, since an escalation of violence would involve hiring newer, less experienced gunmen. As you'd expect the newer gunmen were not the brightest bulbs in the box. But even for dumb cartel gunmen I'm surprised by the sudden increase in accidents and the sudden drop after 2008.

There's also the fact that the police and government authorities are under much more pressure to bring down the number of homicides than the number of accidents. For all these reasons I think it is much more likely that at least some homicides were misclassified as accidents, and that this explains part of the discrepancy during 2007 and 2008.

As funny as I find this, it probably didn't happen in Sinaloa in 2007 and 2008

What I'll do to find out whether the deaths were accidents or homicides is to first ignore the classification specified in the mortality database for all firearm accidents from January 2007 to December 2008. Then use machine learning to see if the characteristics of the original firearm accidents (sex, age, place of occurrence, etc) are sufficient to separate them into homicides and accidents.

Homicides in Sinaloa tend to occur on public streets:

However, accidentals deaths tend to occur all over the place except for 2007 and 2008 when there were a lot accidents on public streets (surprise, surprise):

Once we ignore the firearm accidents that occurred in 2007 and 2008 there are very few accidentals deaths by firearm in Sinaloa, and since the more data the better, I used all 1,834 firearms accidents in the rest of Mexico (excluding Chiapas because its vital statistics are a mess) together with all homicides and accidents by firearm in Sinaloa (excluding 2007 and 2008 obviously) to train a classifier.

There are still some unresolved problems with this approach since the state of Baja California went from having 6 deaths by firearm in 2006, to 100 in 2007, to 6 again in 2008, in comparison homicides went from 303 in 2006 down to 243 in 2007. This could imply that Sinaloa is not the only state with mortality misclassifications and underlines just how difficult it is to get training data (I ignored any possible errors in the rest of the country for classifying the data)

I ended up using the age, sex, place of occurrence of the injury, year of death, marital status and whether an autopsy was perfomed as the variables (including their interactions) in a penalized logistic regression to separate homicides from accidents. To avoid overfitting I separeted 30% of the data into a test dataset.

These are the results of crossvalidating on the test database:

           Reference
Prediction Accident Homicide
  Accident      193       82
  Homicide      166      933
                                          
               Accuracy : 0.8195          
                 95% CI : (0.7981, 0.8395)
    No Information Rate : 0.7387          
    P-Value [Acc > NIR] : 8.951e-13       
                                          
                  Kappa : 0.4942          
 Mcnemar's Test P-Value : 1.360e-07       
                                          
            Sensitivity : 0.5376          
           Specificity : 0.9192         
         Pos Pred Value : 0.7018          
         Neg Pred Value : 0.8490          
             Prevalence : 0.2613          
         Detection Rate : 0.1405          
   Detection Prevalence : 0.2001          
                                          
       'Positive' Class : Accident

Specificity measures the proportion of homicides which were correctly identified, which in this case is more important than misclassified accidents (sensitivity), since I'm trying to prove that the accidents are really homicides. With a specificity of 92% we can be pretty certain that a homicide classified as such is actually a homicide, but with a sensitivity of only 54% we might as well flip a coin when determining how certain we are that an accident is actually an accident. With such a low sensitivity we can consider the number of homicides in the table below as a sort of lower limit.

Year	Imputed Intent	Number of Deaths
2007	Accident	61
2007	Homicide	143
2008	Accident	20
2008	Homicide	178

The result of classifying the 204 and 198 firearm accidents in 2007 and 2008.

As you can see the data looks much more reasonable now, though in 2008 there were still more drug homicides than total homicides, which may have been because some homicides were registered as accidents by a cause other than firearm (in around 23% of accidents the cause is left unspecified). When it comes to violence in Mexico it seems we watch shadows projected on a cave's wall.

Given that the sudden increase in homicides coincided with the end and start of a new year there may have been some sort of law or some legal means that made it ok to classify a homicide as an accident but that is just a matter of calling a tail a leg. But at this point I'm just speculating as to possible motives for recording a homicide as an accident, so I'll stop.

I can't confitm anything. There's something to what you ask, but I can't confirm anything. We have to perfom tests, you'll have the data later. What we know is that it was an accident, but a very strange one because the body was dumped at the public hospital where he died.

—Jesús Antonio Aguilar Íñiguez, Chief of Sinaloa's Investigative Police

No te puedo confirmar nada. Algo hay de lo que preguntas, pero no te puedo confirmar nada. Faltan pruebas periciales por realizar, después tendrás los datos. Lo que sabemos es que es un accidente, pero muy raro porque el cuerpo fue arrojado en el Seguro Social y ahí murió

—Jesús Antonio Aguilar Íñiguez, jefe de la Policía Ministerial del Estado

P.S. The data and code are available from my GitHub

Mexico's most violent cities in 2011

2012-11-26T22:33:00.000-06:00

Note the log scale and that data are incomplete

The INEGI finally released homicide data at the municipality level, analyzing it by state turned out very similar results to the preliminary data released back in August, so I won't repeat the state level charts I did back then, instead I will focus on the trends in violence at the metro area or big municipality level since both Tijuana and Ciudad Juárez had big declines in homicides —though they are still very violent— and Monterrey, Acapulco and Veracruz saw big increases.

As I previously wrote there have been a lot deaths of unknown intent with characteristics similar to homicides since the drug war started, and 2011 was no exception: Coahuila went from registering 2 deaths of unknown intent in 2009 to 216 in 2011, Veracruz went from 197 to 747, Chiapas went from 2 to 614, Tamaulipas had 195 in 2011, and Mexico as a whole went from registering 2,376 deaths of unknown intent in 2007 to 5,630 in 2011.

We can also compare the number of homicides to the number of deaths in the recently cancelled drug war-related homicide database (only including the first 9 moths of 2011) to find out which states had more drug war deaths than total homicides, which in theory should be impossible

State	Year	Drug War- Related Homicides	Homicides
Coahuila	2011	572	478
Sinaloa	2007	414	388
Sinaloa	2008	1084	871
Tamaulipas	2010	1209	1011
Tamaulipas	2011	1108	830

One can also look at the data disaggregated by metro area or municipality (only showing those cities where drug war-related homicides exceeded all homicides by at least 15)

City	Year	Drug War- Related Homicides	Homicides
Culiacán-Navolato	2008	654	460
Culiacán-Navolato	2007	271	131
San Fernando, Tamps	2011	292	159
La Laguna	2011	633	509
Veracruz	2011	261	164
Valle Hermoso, Tamps	2011	95	14
Reynosa	2010	154	92
Miguel Alemán, Tamps	2010	110	58
Mier, Tamps	2010	93	42
Guerrero, Tamps	2010	45	0
San Fernando, Tamps	2010	169	130
Taxco de Alarcón, Gro	2010	75	38
La Laguna	2010	704	673
Mier, Tamps	2011	50	19
Morelia	2011	104	78
Ahumada, Chih	2009	27	4
Guerrero, Tamps	2011	35	12
Gustavo Díaz Ordaz, Tamps	2010	25	7
Jiménez, Chih	2011	47	30
Cosalá, Sin	2011	25	8
Salvador Alvarado, Sin	2008	47	31
~~Sáric, Son~~	~~2010~~	30	14
Camargo, Tamps	2010	30	15

*Saric, Sonora has been crossed out because the deaths were registered in Tubutama

With so many missing deaths I've opted to watermark the plots with a text that says 'incomplete data' since I could probably get better estimates by classifying the deaths of unknown intent, but I need access to the raw data which is not available yet. In the plot at the top of the blog Tuxtla is the safest city, but with 112 deaths of unknown intent and only 10 homicides this is extremely likely to be wrong.

By looking at the metro area homicide numbers you can miss diverging municipality level trends, so for La Laguna and Mexico City I've plotted the data grouped by state.

The model is loess

Since inside a metro area people travel all around the city it is more important to focus on the trend rather than the levels. In Mexico City the increase in homicides has been much more marked in the State of Mexico than in the Federal District, mostly due to trends in Ecatepec and Neza.

In La Laguna violence in the part of the city situated in Coahuila (mainly Torreón) increased while violence in the Durango part decreased (mainly Gomez Palacio)

The model is loes

Cities that saw big decreases in violence:

Cities that saw increases in violence:

Cities where the homicide rate didn't vary much:

And the obligatory map:

Mexico's most dangerous cities for women

2012-11-20T18:54:00.000-06:00

Before the drug war the metro area of Toluca was by far the most violent city in Mexico for females and not Ciudad Juarez as commonly believed, in fact, when cities are ranked by their age-adjusted homicide rates from 1990 to 2006 (right before the drug war started) Juárez was not much more violent than Mexico City (Valle de México in the chart).

To construct the list I used the metropolitan areas as definied by the INEGI plus a made up metro area of the municipalities of Culiacán and Navolato, I also included municipalities like Durango which are not part of a metro area but have a population of more than 400,000.

Since violence in Mexico was declining before the drug war it is also worthwhile to take a look at the yearly rates:

Notice how if we look at the drug war years Toluca actually had a lower female homicide rate than Monterrey in 2010, which used to be the safest metro area in Mexico.

Age-Period-Cohort models and the decline of violence

2012-10-09T19:12:00.000-05:00

Ever since the end of the Mexican Revolution and the Cristero War violence in Mexico inched down in fits and starts from a high of about 60 homicides per 100,000 people to its lowest level sometime during the middle of the last decade (there's some uncertainty about the number of homicides in 2007). Then, the drug war happened and the homicide rate shot straight up.

After reading The Better Angels of Our Nature: Why Violence Has Declined, by Steven Pinker, and looking at reviews of the literature on homicide decline in Europe, it seemed to me as if some of the posited reasons for the decline of violence would involve strong cohort effects —each generation successively becoming less and less violent. This immediately reminded me of Mexico, and so I decided to take a quick look at the data:

Each cohort group is connected by a loess line(only ages 12-60 are shown in the graph)

For a more formal analysis of the decline of violence I used homicide data from the INEGI/SINAIS at the national level from 1985 and at the state/municipality level from 1990 onwards. Since I'm interested in understanding the period before the drug war I discarded all data after 2006. As usual, there are some issues with the data:

About 3% of homicides have no data corresponding to the age of the victim. I'll simply ignore the missing data. The data also presents clumping of victim's ages which end in 0 or 5, to deal with this I grouped the data by 5 year age groups.
Some deaths of unknown intent are probably homicides, for sure I could use multiple imputation to get better estimates of the number of homicides for the age-period-cohort model, but again I'll ignore this.
Previous to 1994 there were a lot of homicides registered in the INEGI database with no year of occurrence recorded. I used the date of registration of the homicide as the year of occurrence before 1994 .
There are some errors in the mortality database where homicides are registered as accidents or simply missing. And of course the Acteal Massacre is missing from the homicide database.

In countries with reliable statistics on offenders most women are killed by men, so I focused on total homicide victims rather than each sex since I see no reason why Mexico should be any different. To explore the data I plotted the age-adjusted homicide rates for each state of Mexico. Since the five states with the biggest declines in homicides all lie in this contiguous area of Mexico I highlighted them in blue.

The border states seem to follow a different dynamic than the rest of Mexico so I plotted the age-adjusted homicide rate for all municipalities whose head locality lies at least 200 km from the U.S. border:

Since we have a bunch of contiguous states which saw big declines in violence and the municipalities near the U.S. border which saw increases, I'll build two different age-period-cohort models for each area using the Epi package (tough I could also use a multilevel model). Prior to building the age-period-cohort model I'll visualize the four classic plots for the two areas of Mexico I mentioned.

The southern states:

Municipalities 200 km from the border:

The top left plot shows the rate versus the age with each birth cohort group connected by a quadratic polynomial regression line. You can see a very clear pattern of each cohort being less violet in the southern states but not in the municipalities near the U.S. border.
The top right plot shows the rate versus the age, with each period connected. You can see that children have low homicide rates and young adults the highest. Near the U.S. border there is a drop in homicide rates after 40 years of age (but remember the cohort effects from the first plot!).
The bottom left plot shows the rate versus the period, with each age group connected. You can see that the decrease in violence occurred primarily among the ages with the highest rates in the southern states but not so much near the U.S. border.
The bottom right plot shows the rate versus the cohort, with the each age group connected. You can see how in the southern states each cohort suffered less violence than the rest, though not so much near the U.S. border (but keep in mind period effects!).

After looking at those plots it seems appropriate to fit an Age-Period-Cohort model to separate the effects of each component:

Age effect: Changes in the rates according to biological process of aging: children have low homicides rates, young adults a high one.
Cohort effect: Changes in the rates according to the birth date of each homicide victim. People born in the 50s were more likely to be murdered than people born in the 90s (well, until the drug war), which could reflect better schooling, higher literacy, etc.
Period effect: Changes in the rates affecting the entire population at a given point in time, related to historical events such as the arrival of the drug cartels to the border during the 90s and the problems with the Familia Michoacana before the drug war.

In the model, I used the 1970 cohort as reference for the age and cohort effect, and 2003 as the period reference since that was the year before the expiration of the assault weapon ban, Chapo Guzmán killing the brother of the Juárez Cartel leader, La Barbie in Nuevo Laredo and the whole La Familia Michoacana thing.

From the chart we can see that the age pattern for homicides follows that found in other countries (Fig 3.3 page 41), once we control for cohort and period effects, with the highest risk of being a homicide victim for people in their early 20s. Interestingly, near the border people remain much much violent throughout their lives. Also note how much more violent southern Mexico was compared to the U.S. border. Infants living in the south where at a higher risk of dying of murder than people in their 20s near the U.S. border (controling for cohort and period effects). This probably because people in their 20s commit infanticide and also kill their peers in their 20s.

Looking at the cohort effect there was a big decrease in violence by each successive generation. The period effect shows a rise in the early 90s which might have been related to the land reform enacted about that time (this is pure speculation) and the heightned importance of the Mexico-US border for traficking cocaine in the 90s, the model also shows a rise after 2003 which is very likely related to the whole breakup of the Federation and the rise of La Familia Michoacana as stated previously, though keep in mind its size compared to the cohort effect (they are on the same scale)

Note: I'm more interested in explaining the decline of violence in Mexico rather than predicting the drug war, so the age specific rates for the 1970 cohort are wrong after age 38 since 2008 was the year homicide rates shot up because of the drug war.

One problem with looking at homicide victims is that we ignore data on those who committed the homicides. But in Mexico there is very little data on those who commit murder, in 1930 the government detained about 6,500 presumed murderers, the same number as in 2011! But in other parts of the world those who commit homicide have similar ages to their victims, in fact they are usually acquaintances (with the obvious exception of infants).

There is also the possibility that rather than a reduction in violence the decrease in homicides could reflect better medical treatment, but Mexico also saw a reduction in all criminal activity:

The trendline was adjusted for seasonal effects and length of month

There is also the fact that the homicide rate in Mexico followed the homicide rates of Canada and the U.S. until the drug war. The consensus is that in Canada and the U.S. people really became less violent.

Data Sources: INEGI, FBI, and Statistics Canada (Criminal Justice)

So my best guess is that each cohort of Mexicans really was less violent than the previous ones. You certainly didn't want to mess with someone born around the time of the revolution, but with each passing generation, for reasons still unknown, people became less and less violent until the late 80s and early 90s saw the birth of the mirrrey generation: people more interested in aping the clothing style of uncivilized proles, with their garrish logos and shirts with legible clothing, than in being violent.

The real tragedy is that what was supposed to be the least violent generation in recorded history is now the protagonist of the drug war:

Each cohort group is connected by a loess line(only ages 12-60 are shown in the graph)

P.S. You can download the code and data from GitHub

Drug use and seizures in the U.S.

2012-09-25T23:59:00.000-05:00

According to data from the National Survey on Drug Use & Health 2001-2011 marijuana and heroin use are up, but cocaine and methampethamine use are down. (Data on methamphetamine consumption was statistically adjusted to account for survey changes)

The spike in heroin users in 2006 is probably because the NSDUH is a household survey and as such it probably has quite a bit of error in capturing the real number of drug users. Since most drug consumption is due to addicts, I'm also showing data for those persons that were dependent or abused drugs.

For drug seizures I used data from the National Drug Threat Assessment 2011 (page 50). The southwest region corresponds to the the US-Mexico border crossings and nearby counties. Basically seizures of all drugs along the US-Mexico border have gone up except for cocaine, probably because cocaine consumption has decreased. (Also see this blog post by Alejandro Hope for Mexican drug seizure data)

As a percentage change I decided to only show seizure data along the northern border for Marijuana and MDMA (ecstasy) since all the other numbers are so small. Also keep in mind that MDMA from Mexico grew from a very low starting point, though seizures are now about the size of those along the Northern border in 2006.

More errors in the Mexican Mortality Database

2012-09-12T21:06:00.000-05:00

I've written before about how the deaths of the Acteal massacre weren't registered as homicides in the mortality database and about the large number of deaths of unknown intent in Ciudad Juárez. There's actually another large error in the database that occurred in January and February of 2007. Lots of accidents of unspecified injury mechanism were registered in the Federal District at the same time there was a never before seen drop in the number of homicides.

The drop didn't just involve homicides but also all kind of accidental deaths, though that is not an error but a drop in the quality of vital statistics data (I only show accidental deaths by cause of transportation in the chart for clarity purposes)

Loking at the data for the whole metro area of Mexico City broken down by state of occurrence one can see that the error only occurred in the Federal District (the spike during late 2007 in the State of Mexico is due transport accidents misclassified as unspecified, though it is difficult to see in the data due to seasonal patterns)

The ages of accidental deaths by unspecified cause that took place during January and February 2007 look different than those that occurred at other times.

The male to female ratio of accidental deaths by unspecified cause that took place during January and February 2007 look different than those that occurred at other times. During January and February more males than females died, just as in homicides and accidental deaths by cause of transportation.

I don't think there can be any doubt that some homicides "leaked" into accidental deaths by unspecified cause. I tried to classify the Jan-Feb accidents into homicides and accidents based on the characteristics of the dead (age, sex, place of occurrence) but given that the cause of death was left unspecified (firearm, suffocation, cut/pierce, etc) I was only able to obtain sensitivities and specificities of .60, which is only a little better than chance.

It looks as if the error didn't just happen in the Federal District but all over Mexico (though obviously no the state of Mexico). You see the same pattern of a big drop in homicides at the same time accidents by unspecified cause rise. Perhaps someone hired his dump brother in law to administer the database and he screwed things up?

The most worrying part is that the error coincided with the first military operations of the drug war. Perhaps the Mexican government saw the drop in homicides and became a little too cocky about its capacity to take on the drug cartels.

In Nuevo Laredo during the first months of 2007 there were more drug related homicides that total homicides according to the INEGI.

Violence along Mexico's Southern Border and Central America

2012-09-07T20:38:00.000-05:00

Rates for Panama and Nicaragua are from 2009, all other countries 2010. Municipalities which are part of a metro area in Mexico are shown with the metro area homicide rate.

Visit the interactive map of homicides

Having just posted on violence along Mexico's northern border, I figured it's time to analyze what is happing south of Mexico where some countries have experienced sharp increases in homicides.

It's a little bit fallacious to compare a country as big as Mexico with its southern neighbors since Mexico's most populous state has more people living in it than the entire country of Guatemala, and all Mexican states are more populous than Belize, also the administrative divisions in each country are a little bit different:

In Mexico the divisions go something like this:

States -> Municipalities (Counties)

In Guatemala:

Deparments -> Municipalities

In Belize:

Districts -> Constituencies

So in the map I'm showing at the top of this post I'm comparing level 1 divisions in Guatemala and Belize (departments and districts) with level 2 divisions in Mexico (municipalities).

In addition, homicide records are not always reliable in both Mexico and Guatemala, in Mexico vital statistics records of homicides based on death certificates are usually more reliable than those compiled from police records, but in Chiapas –which borders Guatemala– police records consistently record a higher number of homicides. Looking at the monthly data one can see that some months both transportation accidents and homicides fall to levels near zero. I interpret this as Chiapas not being able to register every death, being one of Mexico's poorest states this is not surprising. Chiapas health records are not up to par with the rest of the country.

Homicides in Chiapas from different sources:

	2010	2011
INEGI(Vital Statistics)	197	185
SNSP(Police)	513	613

In Guatemala police records consistently document more homicides than vital statistics compiled by health authorities, though there are a lot of deaths of unknown intent. I interpret this as police records being more accurate than vital statistics.

With all that said it's now a matter of collecting the data. Lucky for me, La Prensa Gráfica compiled homicide statistics for Central America and they used the police sources for Guatemala. So, it's mostly a matter of downloading and merging shapefiles to analyze the data:

getMaps <- function(codes, level) {
  ##Different column names to uniquely identify each group
  ##depending on the level requested
  column.name <- ifelse(level == 1, "NAME_0", "ISO")
  ##Download the maps
  country.ll <- llply(codes,
                      function(x) getData("GADM", path = "maps",
                                          country = x, level = level))
  ##Change the id of the maps since some are repeated and we need to
  ##merge them, use ISO code as the unique identifier
  country.ll <- llply(country.ll,
                      function(x) spChFIDs(x, str_c(row.names(x), x[[column.name]][1])))
  ##Merge the list of maps one by one
  for(i in 2:length(country.ll)){
    if(i == 2) {
      map <- spRbind(country.ll[[1]], country.ll[[2]])
    } else { 
      map <- spRbind(map, country.ll[[i]])
    }
  }
  map
}
##Codes for the Central American Countries
country.codes <- c("GTM", "BLZ", "HND", "SLV", "CRI", "NIC", "PAN")
##Get the maps of States/Districts
map <- getMaps(country.codes, level = 1)
##Get the maps of the country outlines
map.borders <- getMaps(country.codes, level = 0)

Here's a map showing the homicide rates from police sources in the Mexican states that border Guatemala and Belize:

Rates for Panama and Nicaragua are from 2009, Mexico from 2011, and all other countries 2010. Data for Central America from La Prensa Gráfica, Mexico from SNSP

If the police homicide statistics are closer to reality, then the border states of Mexico would have a homicide rate of 13 (about the same level as metro Mexico City). The departments and districts of Guatemala and Belize that touch the border with Mexico would have a combined homicide rate of 20 (about the same level as metro Detroit or New Orleans). And as you can see, the most violent part of Guatemala is closer to Honduras and El Salvador than Mexico, with the big exception of Petén. I wonder why the souther border hasn't seen the increase in violence seen in other parts of Mexico? The real contrast is provided by Nicaragua.

The intensity of drug trafficking does seem to be correlated with homicides according to this report by the World Bank (page 13).

For the time series of homicide rates I added a line representing the Mexican states of Baja California, Sonora, Chihuahua, Sinaloa, Durango and Nayarit to serve as comparison, together these states would have a population similar to that of Guatemala and about twice that of El Salvador or Honduras, though their area is much bigger and their GDP per capita much higher. Belize is a pretty small country, so keep in mind that while the trend is increasing, the number of homicides went from 41 in 2000 to 129 in 2010. Another thing to keep in mind is that Central American countries are very young and Ceteris Paribus having more young adults will mean more homicides.

Sources: UNODC Homicide Statistics 2012, INEGI

Violence in Guatemala as of 2011 is at its lowest level since 2004. But homicide rates are higher in all Central American countries compared to 2000:

Sources: UNODC Homicide Statistics 2012, INEGI

Here's an interactive map:

Municipalities which are part of a metro area in Mexico are shown with the metro area homicide rate.

P.S. Data and code are available from GitHub

Homicides along the United States–Mexico border

2012-09-04T18:37:00.000-05:00

The drug war in Mexico has increased the homicide rate in several parts of the country, but one of the most affected regions has been those areas near the U.S. border.

Sources: CDC / INEGI. For Mexican municipalities the distance to the border was measured from the main locality of each municipality, for US counties the distance to the border was measured from the centroid.

Violence has generally stayed on the Mexican side of the border. The trend in homicides for all U.S. counties at least 160 km from the Mexican border has been downward since the drug war started.

Sources: CDC / INEGI

Keep in mind that violence in Mexican municipalities at least 160km from the border was lower in 2006, when the drug war started, than in 1999.

P.S. When referring to homicides in the US I included those deaths that were the result of legal interventions since in Mexico police/military interventions are usually counted as homicides.

Preliminary homicide data for 2011

2012-08-20T22:32:00.001-05:00

The Mexican government through the INEGI just released preliminary homicide data for 2011. I feel bound to remind my readers that the government also released preliminary data in 2010, with the state of Chihuahua missing over a thousand murders, so the data should be interpreted as strictly preliminary.

According to the INEGI the homicide rate was 24 and the total number of homicides in 2011 was 27,199.

Homicides actually went down in many of the most violent states in 2010. Chihuahua, Sinaloa, Baja California and Sonora all registered a drop in homicides in 2011, though they still are way above their longtime trendlines. In Chihuahua and Baja California it's been claimed that part of the reason for the drop in homicides is the fact that the Sinaloa Cartel won control of the plaza (the drug smuggling routes and the political and law enforcement arrangements through which drug traffickers work). Since the Mexican government seems to be going after the Zetas rather than Sinaloa there seems to be one hell of an incentive to invade other cartel's plazas.

Keep in mind that Tamaulipas had lots of deaths of unknown intent in 2010 and it is possible the same thing happened in 2011.

The rise in homicides in Michoacán and Guerrero used to justify the drug war now seem quaint and barely visible

Veracruz in spite of all the bad press Veracruz doesn't seem to be all that violent relative to other regions of Mexico, though I wonder how it breaks down at the municipality level? Without access to the mortality database I have no idea. According to the drug-war related homicide database the Veracruz metro area is extremely violent.

The rate of increase has been higher for the State of Mexico than the Federal District, it would be interesting to break it down for the Mexico City metro area.

The data is not without problems, for example in Chiapas the vital statistics system doesn't seem to be very good, with falls in both homicides and transportation accidents coinciding with the New Year.

The number of homicides in the the SNSP database was greater than the number of homicides according to the INEGI in San Luis Potosí . This is very likely due to the high number of deaths of unknown intent, but I need access to the 2011 mortality database to be certain.

Deaths of unknown intent and by homicide in San Luis Potosí

ANIODEF	Homicide	Unknown
2004	135	36
2005	131	66
2006	160	68
2007	145	3
2008	200	11
2009	209	17
2010	357	99

Machine learning for better homicide counts in Ciudad Juarez

2012-07-30T07:28:00.000-05:00

Photo Credit: Jesús Villaseca Pérez

Ever since March 2008 Ciudad Juárez began to register an alarming number of homicides becoming Mexico's most violent city. According to the Mexican vital statistics system Ciudad Juárez (coterminous with the Juárez municipality) went from having just 202 murders in 2007 to 1,616 in 2008, 2,397 in 2009, and 3,686 in 2010.

Mexican and US officials explain the dramatic increase in violence as due to a conflict between the Sinaloa and Juárez Cartels. After a new governor was elected in October 2010 Ciudad Juárez does seem to have started turning around, but it is still an extremely violent city.

Mortality from ill-defined conditions is quite high in Ciudad Juárez. Deaths of unknown injury intent went from 33 in 2004 to 193 in 2010. It is an open question just how good the homicide records are in places that saw incredible rises in homicides since one can only assume that forensic services were overwhelmed and that educated professionals like doctors were the first to leave town.

Recently there was a story in the New York Times about how Target figured out a teen girl was pregnant before her father did based on her shopping patterns, and if Target can classify its shoppers into pregnant and not pregnant based on the stuff they buy, I can certainly train a computer to classify deaths in Mexico based on the characteristics of the deaths.

Intuitively if someone were to ask you to guess the type of death of a 70 year old woman whose cause of death was transport related you would probably guess it was an accident. On the other hand, if you had to guess the type of death of a young adult male whose cause of death was a firearm, and the injury took place on a public street in Juárez, you would probably guess it was a homicide.

The Mexican government supposedly keeps a record of all deaths in its mortality database. For my purposes there are four types of violent or injury intent deaths:

Accidents (unintentional injuries)
Suicides (self inflicted injuries)
Homicides (intentional injuries)
Unknown Intent (cases where forensic or legal experts determined information was not sufﬁcient to make a decision about the injury intent)

There is a fifth type of injury intent death: "Legal intervention, operations of war, military operations, and terrorism" but there are only about 30-40 deaths of this type in Mexico each year. Most of them occur in Puebla. It is very likely that most of the shootouts where the police or military kill someone (or die themselves) are classified as plain old homicides, so I chose to recode this type of death as homicides.

The Mexican vital statistics database assigns every death an International Classification of Diseases (ICD) code which is used worldwide to categorize diseases, injuries, and external causes of injury. Sadly, it can be a little too specific since it includes distinctions for "wounds inflicted by macaws" and "wounds inflicted by parrots," in addition it combines the injury mechanism (parrot, macaw, drop bear, chupacabras, etc.) with intent (suicide, homicide, etc.), e.g., the code for accidental death by handgun is "W32," and the code for homicide by handgun is "X93".

Ideally I would have a way of classifying the injury mechanism into meaningful groups separate from intent, something akin to the game of clue where there are half a dozen or so weapons like candlestick, knife, rope, gun, etc. that can be used to harm people, except since epidemiology is a real science we would have to substitute the name of the weapons for fancy names like "struck by or against," "cut/pierce," "suffocation," "firearm," etc. Lucky for me the CDC's National Center for Health Statistics has already done exactly that with its External Cause of Injury Mortality Matrix for ICD-10

Once we have recoded the mortality database to include the mortality matrix we can visualize the different types of violent or injury intent deaths according to injury mechanism:

You can visually notice the resemblance between homicide deaths and deaths of unknown intent since both tend to involve firearms. Deaths by accident are mainly caused by transportation (motor vehicles) and suicides by suffocation (think hanging).

It's also worth pointing out that accidental deaths by unspecified mechanism increased at precisely the same time homicides shot through the roof, which could imply that there was some leakage of homicides into accidents, but in this post I'll go with the conservative assumption that all homicides, suicides and accidents were correctly classified by the Mexican health authorities.

The injury mechanism isn't the only useful information we can use to differentiate the type of death since victims of homicide also tend to be younger. We can also use the location where the body was found and the year of death. I could certainly use a more complex model that included marital status, day of week when they death occurred, and so on, but with the high specificity and sensitivity that resulted from the simple model I used, I saw no need to complicate things further (that and my laptop has 2GB of RAM).

Of course things are never as simple as in textbooks, and the Mexican mortality database has some missing values. I assumed homicides where the year of occurrence was not available occurred in the same year the death was registered. For the rest of the data I used k-Nearest Neighbors to impute the missing values.

Most accidents by unspecified injury mechanism were classified as transportation accidents which agrees with previous findings that road deaths are under-counted in Mexico.

The cleaned up dataset looks like this:

EDADVALOR         CAUSE SEXOtxt     LUGLEStxt ANIODEF PRESUNTOtxt
7.483315 All Transport    Male          Home    2007    Accident
5.656854       Firearm    Male          Home    2007    Homicide
6.164414       Firearm    Male Public Street    2009    Homicide
6.082763       Firearm    Male          Home    2007    Homicide
4.690416     Poisoning    Male Public Street    2004    Accident

Where:

EDADVALOR is the square root of age in years
CAUSE is the injury mechanism
SEXOtxt is the sex of the victim
LUGLEStxt is the place where the lesion occurred
ANIODEF is the year the death occurred
PRESUNTOtxt is the injury intent

I divided the dataset into training (75% of the data) and test sets. I then fit a penalized a regression (glmnet package), a support vector machine (with a radial kernel since it had better performance than a linear one), and a random forest model to the training set and evaluated their accuracy against the test set using the caret package.

The random forest model had the highest accuracy (though within the margin of error of the SVM) and that's the algorithm I used to classify the deaths. Here's its confusion matrix:

Confusion Matrix and Statistics

          Reference
Prediction Accident Homicide Suicide
  Accident      743       50       5
  Homicide       92     2083      22
  Suicide        19       33      92

Overall Statistics
                                          
               Accuracy : 0.9296          
                 95% CI : (0.9201, 0.9383)
    No Information Rate : 0.69            
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.8422          
 Mcnemar's Test P-Value : 4.468e-05       

Statistics by Class:

                     Class: Accident Class: Homicide Class: Suicide
Sensitivity                   0.8700          0.9617        0.77311
Specificity                   0.9759          0.8828        0.98278
Pos Pred Value                0.9311          0.9481        0.63889
Neg Pred Value                0.9526          0.9119        0.99098
Prevalence                    0.2721          0.6900        0.03791
Detection Rate                0.2367          0.6636        0.02931
Detection Prevalence          0.2542          0.6999        0.04587

The random forest had a sensitivity of 96% for homicides, that is, out of the 2166 homicides in our testing sample, 2083 where classified correctly. And a specificity of 88%, that is, out of the 973 suicides and accidents in our testing sample, 859 where classified correctly.

Here's the comparison with the original data:

Year	Injury Intent	Original Deaths	Imputed Deaths
2004	Homicide	208	226
2005	Homicide	275	288
2006	Homicide	221	248
2007	Homicide	202	300
2008	Homicide	1616	1679
2009	Homicide	2397	2476
2010	Homicide	3686*	3867*

* Under-counted by about 3%

The imputed number is still not the final number of homicides since (ignoring any statistical error and clandestine graves) homicides are under-counted by around 3% for the last year because the database has a cutoff date for registering deaths of December 31st. Taking this into account and rounding up, the number of imputed homicides in Juárez woud be close to 4,000. According to the SNSP there were 3,903 homicides in the state of Chihuahua (yeah right), and according to the criminal rivalry database there were 2,738 drug war-related homicides in Juárez during 2010.

Total 2008-2010	Original Deaths	Imputed Deaths
	7699	8022

Why did it happen?

The high number of deaths of unknown injury intent that resemble homicides could be due to several factors:

The takeover of law enforcement functions by the military in 2008, then the federal police in 2010, and yet again the municipal police in 2011 could have played havoc on record keeping.
The high levels of violence overwhelmed forensic services and legal authorities.
Some estimates put the number of people who left Juárez as high as 230,000, this surely that had some effect on the quality of vital statistics.
Given that classifying deaths of unknown intent increased the number of homicides by about 50% in 2007-- before the violence started-- statistical manipulation is certainly worth some consideration. According to wikileaks Juárez had 316 murders in 2007 (though it's not clear if the number refers to the Zona Norte) which is close to the imputed estimate.

Future Research

Juárez is not the only place where deaths of unknown intent increased, it's also worth checking out states like Tamaulipas, Coahuila, San Luís Potosí and Durango.

Sometimes accidental deaths by firearm have interesting patterns

In Michoacán homicides dropped soon after the start of the Operation Michoacán on December 11, 2007, but accidents of unknown injury mechanism rose at the same time, since transport accidents dropped at the same time as homicides it is not entirely clear what happened:

P.S. You can download the code and data at my github account

Poll of polls including the 'quick count'

2012-07-02T19:24:00.000-05:00

As expected Peña Nieto won and AMLO came in second place, but the polls way overestimated the voting intention of Peña Nieto, underestimated AMLO's expected vote, and to a lesser extent also underestimated Josefina Vázquez Mota's vote.

The Federal Elections Board in Mexico runs a "quick count" (conteo rápido), a random sample of returns from voting booths across the country, which serves as a highly accurate exit poll. Measuring the euclidian distance from the normalized "quick count" to the voting preferences, the most accurate pollsters were: SDP Noticias-Covarrubias, Grupo Reforma, Ipsos-Bimsa, and UNO TV-María de las Hera. The worst performing pollsters were Milenio-GEA ISA and Indemerc.

Rerunning my polls of polls using the quick count results as if they were the result of a poll taken on election day with a massive sample size and free of pollster bias (I plan on rerunning this analysis when the final election results are known) I get the chart at the top the post. The results are quite striking, particularly with regard to Pena Nieto, and indicate a relatively big bias among most pollsters. I'm very curious about what it is the polling firms did wrong to oversample PRI voters or otherwise bias their results.

There's always the possibility that AMLO had a late surge, but it doesn't look like there was one since some of the private exit polls also tended to put Peña Nieto ahead of what the "quick count" predicted. And exit polls are usually much more accurate than polling. It's very likely there was bias.

Reference

Simon Jackman. Bayesian Analysis for the Social Sciences. John Wiley & Sons, 2009

Final poll of polls

2012-06-27T17:23:00.001-05:00

On Sunday, July 1, 2012 Mexican voters will go to the polls and elect a new president. In Mexico whichever candidate wins a simple plurality of the national vote is elected. Before the election of Vicente Fox of the PAN in 2000 Mexico had been ruled by a single party (the PRI) for over 70 years.

There are three main parties in Mexico:

The Partido Revolucionary Institucional (PRI) commonly described as center-left. Its candidate is the former governor of the State of Mexico Enrique Peña Nieto (EPN)
The Partido Acción Nacional (PAN): The right-wing party which currently holds the presidency. Its candidate is Josefina Vázquez Mota (JVM)
The Partido de la Revolución Democrática: A left wing party which lost the presidency in 2006 by the razor thin margin of .56%. It's candidate is the same one as in 2006, Andrés Manuel López Obrador (AMLO)

The Data

My data comes from the poll of polls by ADN Político. I made some adjustments to the data:

I'm using the midpoint of the start and end dates of the surveys as the polling date
I added a couple of polls conducted by Indemerc and corrected some missing polls from El Universal - Buendía y Laredo
I'm using weekly averages of the GEA-ISA daily tracking poll because it has some weird periodical artifacts

The Model

My first idea was to simply model the presidential race using a dirichlet regression for all four candidates smoothed with splines. This ignores any effect each polling firm may have (some polling firms tend to favor one particular candidate)

I later switched to a Bayesian method of pooling the polls. I'm using a Kalman Filter that corrects for House Effects (the bias of each polling firm) as described in Pooling the Polls Over an Election Campaign (pdf), by Simon Jackman (any mistakes are obviously mine). The only difference from the paper (a minor one) is that since Australia has two main political parties and Mexico has 3 main ones plus one small one, the precision is calculated a bit differently.

I could certainly have improved the forecast by taking into account the economic and security conditions in Mexico since those are the two issues which come out on top when voters are asked what they care about the most. But with virtually only one free presidential election having happened in Mexico (the 2006 one doesn't count since it was very close) it would involve more time than I'm willing to commit.

Given the security and economic situation in Mexico, once campaigning started Josefina Vázquez Mota dropped in the polls. Even former president Vicente Fox recently stabbed the PAN in the back and called on the electorate to vote for the front-runner Peña Nieto and accused the current president of violating human rights due to the drug war.

Forecast

Mexico enforces a polling ban three days before the election (starting tomorrow) and there seems to be very noticeable trend in the voting intention of the candidates so I simulated draws from the posterior voting intention of each candidate and then used the auto.arima function from the forecast package to predict what the trend would be on election day.

House Effects

The model could also be improved by using better priors for the house effect of each polling firm. For example, if you believe there is an institutional bias in favor of Peña Nieto and that the most thrustworthy pollster are Reforma, SDP Noticias and UNO TV then you'll disagree with the estimate from the model I'm using with diffuse priors.

The three polling firms where Peña Nieto polled the worst where Grupo Reforma, SDP Noticas-Covarrubias, and UNO TV-María de las Heras (María de las Heras was the worst pollster at predicting the 2006 election). All three polling firms reported and increase in Peña Nieto voting intention from May to June (Uno TV 39% -> 40.1%, SDP Noticias 40% -> 41%, and Reforma 38% -> 42%)

It would take an act of god for Peña Nieto not to be declared the winner on Sunday. The probability of AMLO beating JVM for the second place is 96%.

Interactive Chart

I may still be missing one or two polls in this analysis so be sure to visit my interactive chart for the latest info and forecast.

One thing is for certain: there is no stopping Enrique Peña Nieto; the PRI will soon be here. And I for one welcome our new prehistoric overlords. I'd like to remind them that as a trusted blog personality, I can be helpful in hiding crime statistics (under the bed) while the proletariat toils in their underground meth labs.

References

Pooling the Polls Over an Election Campaign, Australian Journal of Political Science, 2005 V40(4):499-517

Elect-o-meter

2012-06-27T07:39:00.000-05:00

Check out the interactive poll of polls I made.

P.S. There is a poll ban starting tomorrow so I'll restrict access to the interactive chart to people outside Mexico.

Campaign Effects

2012-06-22T00:00:00.000-05:00

In Mexico presidential campaigns last 90 days. This year campaigning started on March 30, before this date all campaigning was prohibited. Judging by the way Josefina Vázquez Mota's support fell after she started campaigning voters either didn't like what they saw in her ideology and issue stance or she is a very bad campaigner.

Debate Effects

2012-06-21T09:00:00.000-05:00

Gabriel Quadri, the candidate for the Mexican presidency of a small party practically owned by the teachers' union must be a good debater. Since his debate performance he has managed to consistently poll over the 2% required for his party to keep its registration. One small problem with this analysis is that sometimes polling firms report voter preferences as rounded numbers, and for candidates polling in the 0-4% range this introduces some artifacts.

Poll of polls - June 20

2012-06-20T21:57:00.001-05:00

Time for another update of my poll of polls. With less than two weeks remaining until voting day the race is on! I mean the race for second place because Enrique Peña Nieto of the Partido Revolucionario Institucional has pretty much cinched the presidency for the next six years.

I've now switched to a Bayesian method of pooling the polls. I'm using a Kalman Filter that corrects for House Effects (the bias of each polling firm) as described in Pooling the Polls Over an Election Campaign (pdf), by Simon Jackman. I'm using diffuse priors for the house effects but I could probably switch to using the posteriors for each polling firm from the 2006 election, if I ever get around to modeling it.

The thing that worries the most is that polling firms tend to be highly variable, for example, in 2006 the most accurate pollsters were Reforma and GEA-ISA, but in the last elections they did not perform so well. Oh well, we'll see what happens.

To recap from previous blogs

My data comes from the poll of polls by ADN Político.
I added a couple of polls conducted by Indemerc and corrected some missing polls from El Universal - Buendía y Laredo
I'm using weekly averages of the GEA-ISA daily tracking poll because it has some weird periodical artifacts

References

Pooling the Polls Over an Election Campaign, Australian Journal of Political Science, 2005 V40(4):499-517