Crimes and Communities: statistics and prediction

2013-02-22T21:13:00.000+01:00

The code source of this blog

http://code.google.com/p/the-criminality-usa/

Pennsylvania expenses and Chester city criminality

2013-02-01T13:04:00.001+01:00

As we can see in this picture the evolution of the total expense in Pennsylvania follow the curve of the violent crimes in Chester City . The Health and human services is the cause of this principal expense.

The curve of the protections of person is following the violent crimes. But in 2011 the decreased expense has created certainly the growth of the violent crime of the same year.

Predict violent crime for 2013 at chester city , PA

2013-01-09T11:17:00.001+01:00

This is a study of crime prediction at chester city, PA.
Prediction is an hard task. With a low data content the must is to exponential method to predict the violent crimes rate.
For 2013 , the violent crimes should decrease lowly .
It is a tendency.

Predicted crimes value by 100k hab are

	Real value for 2012	Predict for 2013
Violent crime	3174	3165.4009559846
Murder and nonnegligent manslaughter	64	64.2627077813
Forcible rape	50	51.7887222942
Robbery	637	635.089397512
Aggravated assault	2423	2414.2601283972

Chester city criminality evolution

2013-01-08T13:14:00.000+01:00

I continue my study with chester city in pennsylvania. A very hard city where the criminality is crazy.

I give the violent crimes and property crimes values extracted from FBI and Pennsylvania Uniform Crime Reporting Program.

We can see an long evolution of the criminality during this last decade, apparently more effort are made in year 1999 and 200 to resolve some issues in security. but the crime is increasing since this period.

Brut data

	2012	2011	2010	2009	2008	2007	2006	2005	2004	2003	2002	2001	2000	1999	1995	1990	1985	1980
Violent crime	3174.882629108	3148.4741784038	2441.8355312534	2647.2118043199	2879.5239914843	2568.0045653415	2652.9346741797	2524.9384985537	1968.2813215219	1491.2918248136	1847.8495785606	1540.4643089607	3022.7383730396	3178.8539264549	5484.1647273095	5191.6093272171	3107.1683309558	3216.278306531
Murder and nonnegligent manslaughter	64.5539906103	61.6197183099	64.3293663557	38.3257138164	46.399912659	73.3715590098	48.5292928204	40.5504041524	51.1591588357	18.8430374976	51.32915496	35.2571056628	51.5547837412	22.3164472216	55.6511180816	35.8371559633	15.6027104137	35.0071108194
Forcible rape	49.882629108	67.4882629108	83.0920982095	68.4387746722	114.635078334	67.9366287127	56.6175082904	91.914249412	83.4702065214	88.8314624889	83.747568619	43.3933608158	70.5486514354	128.9394728359	204.8972983912	222.1903669725	187.2325249643	126.9007767203
Robbery	636.7370892019	627.9342723005	552.1603945534	531.0848914561	646.8693705988	627.7344493057	649.7533094282	619.0695033927	457.7398422144	288.029287464	437.6485843959	368.8435669343	697.3462853422	662.0546009075	1558.2313062835	1017.7752293578	815.7988587732	1207.7453232688
Aggravated assault	2423.7089201878	2391.4319248826	1742.2536721347	2009.3624243752	2071.6196298925	1798.9619283133	1898.0345636408	1773.4043415966	1375.9121139503	1095.5880373631	1275.1242705857	1092.9702755478	2203.2886525208	2365.5434054899	3665.3850045533	3915.8065749236	2088.5342368046	1846.6250957226
Property crime	3606.220657277	3879.1079812207	3355.8486115578	3556.0787319664	3788.416398275	3614.2286475176	3957.8334366828	4000.9732096997	3473.437626215	2543.810062182	3317.4843311001	3221.9570405728	4862.430129701	5482.4072007736	8714.4591723161	7530.5810397554	6851.8188302425	7679.6849360026
Burglary	1364.4366197183	1625.5868544601	1249.0618634073	1335.9248816009	1228.2329821497	1187.5322698986	1197.0558895689	1297.6129328756	1066.2645736288	619.1283749226	645.6667387076	659.0366673899	1240.0282194606	1522.4776215627	2592.8361833451	2119.1704892966	2342.6355206847	3369.4344163658
Larceny- theft	1866.1971830986	1836.8544600939	1710.0889889568	1730.1322237127	1948.7963316775	1725.5903693035	1995.0931492815	1919.3857965451	1658.6337812004	1152.117149856	1745.1912686406	1659.7960512042	2243.9897975796	2534.1565622753	4062.5316199535	3741.3990825688	3586.3944365193	3435.0727491522
Motor vehicle theft	299.2957746479	416.6666666667	396.6977591937	490.0216266528	611.3870844478	701.1060083154	765.6843978324	783.974480279	748.5392713859	699.8842499125	842.8787551329	816.3376003471	1191.1868453899	1157.9756502765	2059.0913690175	1670.0114678899	922.7888730385	875.1777704846
Arson	76.2910798122	76.2910798122	88.4528787391	109.5020394755	81.88219881	84.2414196038	67.4017955838	97.3209699657	72.6998572929	72.6802874909	83.747568619	86.7867216316	187.2252672709	267.7973666592

Improving house can decrease the violent criminality

2013-01-06T22:52:00.001+01:00

Introduction

The subject of the study has permitted to obtain a linear model. A model is entirely good to test or stress a system. In this idea , I propose some test suggestions.
I choose to follow the city of Chester in Pennsylvania that is our maximum value for the violent crimes in USA and the mean of all the violent crimes in USA.

In our study,the actual violent crimes of chester city is 4877 for 100k habitant .It should be different today.
Our model estimate the value to 3916 violent crimes. An error of 20%
Poverty and crime rose as the chester city declined since 1950.
The racial makeup of the city was 17.2% White, 74.7% Black, 0.4% Native American, 0.6% Asian, 0.1% Native Hawaiian, 3.9% of some other race, and 3.0% from two or more races. 9.0% were Hispanic or Latino of any race.

The model estimate the mean of the violent crimes in USA to 1174 violent crimes to 100k habitant

Some question could be given to stress our model .

Immigration reducing

The first I propose : is reducing recent immigrant a solution ?
If we raise 10% of the recent immigration the model give this result :
on chester city the violent crime appear stable to 3917 violent crimes
on the usa the mean increase to 1186(+1%) violent crimes per 100k.
Recent immigration appears not a problem for the security.

In second : Is reducing all immigrant generation a solution ?
Chester city decrease the violent crime to 3915 violent crimes per 100k .
On the mean in USA the violent crime decrease to 1171 violent crimes.
The result is not signifiant.

The idea of reducing immigration is not a good idea and not offers a good result.

Education

Is giving a better education a solution ?
If we decrease by 10% the not graduate or less graduate the model is offer this result:
Chester city increase their violent crimes value to 3922.
And the mean violent crime in USA to 1181.

The result is not corresponding on the simple idea that a better education can really change the given actual criminality.

Ecthic

Is increasing ethnic mix a solution ?
If we increase the white population by 10% , the model is offering this data:
for chester city : 3916 violent crimes . No impact on the criminality.
for mean in USA : 1166 violent crimes

As other result,we can improve the criminality in adding more white population.

Housing

Is decreasing house boarded a solution?
House boarded is a ravage in a city. Offering inglorious area and give bad image on the land this house need to be cleaned.
If we decrease by 10% the house boarded , the criminality decrease :
for chester city we obtain 3888 violent crimes per 100k that is -0.7%
for the mean in USA the value decrease by 0.5% at 1168 violent crimes

Employment

Is increasing employment a solution ?
The employment permits a more great stability on a family.
if we increase the employment by 10%, the criminality increase:

for chester city we obtain 3946 violent crimes per 100k that is +0.8%
for the mean in USA the value decrease by 5,3% at 1235 violent crimes

It s an interesting point to save . How to explain if we increase the employment the criminality increase. The model don t explain some relation between the employment and the violent crimes.

Mathematical approach

I have an other approach , more mathematic. For example we can search some extrema in our model.

I start in reviewing the linear model and finish in testing each value independently.

Best regression values

There is two extrema value in our model. A positive value that if we decrease it should decrease the violent crimes and a negative value that inversely do the same thing.

For the positive value, the name is OwnOccMedVal that corresponds to the owner occupied housing on median value. If we decrease this value on all county or in cherster city the result decrease.
For chester city the criminality decrease to 3902 (-0.35 %)
For the mean in USA the value is important and decrease to 1084 violent crimes by 100k hab (-7,60%).

For the negative value, the name is numbUrban that corresponds to the number of people living in areas classified as urban.
If we increase this value by 10 pourcents, the violent crimes decrease on chester city and in the USA.
for chester city the value decrease to 3895 violent crimes (-0,52%) that is better that the previous value.
fro the mean in the USA the result is 1155 violent crimes and decreasing by 1,60%.

Testing each variables independently

Testing value by value is more mechanic but permit to search between the model on the data which value to change to obtain a best result.

I have tested in increasing each value by 10 %
I obtain two values , one for chester city and one for the mean in the USA.
Fo chester city the value is PctLargHouseOccup corresponding to the percent of all occupied households that are large (6 or more people). We decrease the criminality to 3786 crimes (-3,31%).
For the mean of the USA is PctPersOwnOccup corresponding to the percent of people in owner occupied households. We decrease to the principal value to 1016 (-13,43%) .

In testing in decreasing each value by 10% we obtain different value for chester and the mean value in the USA.
For chester city the more impacting variable is racepctblack correspond to the pourcent of black people in the city. In chester city the number is more than 74%. In decreasing this value by 10 pourcent we can decrease the violent crimes to 3757 (-4,05%).
For the mean in the USA the value is PersPerOccupHous corresponding to the mean persons per household. In decreasing this value by 10% the result is decreasing to 1048 (-10,72%).

Conclusion

It s difficult to give a valid conclusion on the fact that the most important variables that is apt to decrease the violent crimes are based on house. It ' s easy to imagine that in giving more possibility to be owner of their house we can improve the socio economic environment . But to enhance this need , the population have to get a better economic situation that is not the case at Chester city.
In chester city, reducing black concentration and increasing the house size could affect durably the criminality. It s not impossible to politics to take action on this parameters.

Four groups: a visual point of view

2012-12-30T22:37:00.000+01:00

I follow the study written on the message the groups of my study article

We start with the first screen that colorize my first group explained. This first group describes the most hard and poor people where we found violent crime problems.

We can see that the value is shared on the entire map. Certain states are more represented as California New York, Michigan, Delaware and Kansas

Group around the violent criminality

The fourth group the group of the ideal family , white with two children ,.. This perfect group is shared along the united states of america. More representative states are based on the north . Where the south is more sweet.

The group of the ideal family

The group of workers. We found this group in all the state with an equal representation.

The group of poor workers

The group of manager is really interesting. because the structure of this group is really important in California , New Jersey, Connecticut and Massachusetts . Idaho is lowest represented as Kentucky , Louisiana, ...

The group of the managers

Images of violent crimes by state in USA

2012-12-30T19:26:00.000+01:00

Images of violent crimes by state in USA

The first image propose the max county value of the state
The second image is the mean value of the violent crime of the counties

From the max violent crimes on county

From the mean violent crimes of the counties

Predict and model estimation

2012-12-30T12:09:00.000+01:00

Short summary

For study and to estimate a good model we need a scientific approach in segment values to construct the model and values to tes the model.
For certain method as neural network , we need to add some validation values to stop and find the stability of the engine.

Our exercise is to try to extract from the existing data , a stable model to predict the value of the violent crimes for 100k habitant .The typology of the violent crimes is large each county communities and countries have their proper approach. We can include murders, sexual acts, ...
More approach is considered on my study by regression , SVM and neural network.

The values segmentation

The data are cut as this:

1094 values for the model estimation
401 values for the validation of the model
101 values to verify and test the precision of the model

The first strategies

The data are really hard to be separated , it s why in ours strategies the classification could be short.

The first analysis should permit to estimate models in a classical manner.
A first good point to compare with strategy of data reduction.
A second analysis should cut the data in more than 2 classes and models. Yes , to have a more accuracy estimation we can cut our model and use a SVM for automatic classification on new values.
A third approach is to remove unnecessary data as extrema or non connected individuals or variables.
The last study should mix these strategies.

The groups resulting of my study on classification

2012-12-30T00:22:00.001+01:00

Short intro

I want to present the final elements resulting of the review of the variables of my study on violent crimes.
Why grouping variables ? It is a good question but we need to know how variables interact and coexist . At the same time their understanding permit to implement some strategy on research as vectorization or map reducing.
When I have started my study I haven't searched to define group of people. But the study has moving me to identify some humans group. When I say humans group , I want to talk about social relationship.
It s really a surprise for me to distinguish clearly social and cultural group.
I let you discover my final analysis.

Results

The first group : grouped around the violent crime per 100k hab variable

HousVacant, LandArea, LemasPctOfficDrugUn, numbUrban ,NumIlleg ,NumImmig, NumInShelters, NumStreet, NumUnderPov, PctForreignBorn, PctHousNoPhone ,PctIlleg, PctLargHouseFam, PctLargHouseOccup, PctLess9thGrade , PctNotSpeakEnglWell , PctPersDenseHous, PctPopUnderPov ,PctRecentImmig ,PctRecImmig10,PctRecImmig5,PctRecImmig8, PctVacantBoarded , PctWOFullPlumb , pctWPubAsst , PopDens ,population , racepctblack ,racePctHisp , ViolentCrimesPerPop.

I know that the readability of this group is not easy. But I can give some information.

This group is a group of people living in an area where house are vacant and/or boarded, really urban and very dense, with illegitime children, immigrant, where people don t have a phone, living in large house with public assistance and not graduate. Black and hispanic race.

We have all the principal values around the violent criminality. Reducing one of this factor can have a real impact on crime activity.

The second group : the poor workers

agePct12t21,agePct12t29,agePct16t24,agePct65up, FemalePctDiv, householdsize, indianPerCap, MalePctDivorce, MalePctNevMar , MedOwnCostPctInc ,MedOwnCostPctIncNoMtg , MedRentPctHousInc ,MedYrHousBuilt , PctEmplManu, PctEmplProfServ, PctHousLess3BR, PctImmigRec10,PctImmigRec5,PctImmigRec8 ,PctImmigRecent , PctNotHSGrad, PctOccupManu, PctUnemployed, PctUsePubTrans, PctVacMore6Mo, pctWFarmSelf , pctWSocSec, PersPerFam,PersPerOccupHous,PersPerOwnOccHous,PersPerRentOccHous , racePctAsian, TotalPctDiv
This group correspond to the mean of the population that use public transport, have manual work or unemployed , without social security, immigrant , without diploma. Indian and Asian are represented.

The third group : the managers

AsianPerCap ,blackPerCap, HispPerCap, medFamInc ,medIncome ,MedNumBR , MedRent, OwnOccHiQuart ,OwnOccLowQuart, OwnOccMedVal , PctBSorMore, PctOccuptMgmtProf, perCapInc, RenLowQ,RentHighQ,RentMedian, white per cap
This group is interesting, because we mix some race as white, black , asian and hispanic . An this group is composed by managers whose living in their proper house or renting it. We can say that this group manage the second group.

The fourth group : the ideal family

PctBornSameState, PctEmploy, PctFam2Par, PctKids2Par, PctSameCity85 ,PctSameHouse85,PctSameState85,PctSpeakEnglOnly,PctTeen2Par , pctUrban, pctWInvInc, PctWorkMom,PctWorkMomYoungKids , PctWRetire, pctWWage ,PctYoungKids2Par , racePctWhite , PctHouseOccup ,PctHouseOwnOccup , PctPersOwnOccup
This incredible group is a perfect family as we can see in the idealiste literature.
With two kids, living in the same area since a long time , speaking in english , working or retired and not unemployed and white race.
Stability of the group on their area permit the employment and the tv dict some idea as two children by family, ...

French study

2012-11-19T13:49:00.001+01:00

PDF French Study

Predict by mixing strategy

2012-11-19T13:41:00.001+01:00

My last prediction analysis is

Method	RMSE	MAE	MSE	ARV
likelihood, gaussian mixture	0.1241	0.087554	0.015401	0.42164
Full data set	0.13475	0.098797	0.018157	0.49709
Full data set cut in 2 classes	0.13686	0.097323	0.01873	0.51276
Full data set cut in 3 classes	0.13521	0.094228	0.018282	0.50052
Removed Variables	0.13406	0.097274	0.017972	0.49202
Removed Communities	0.12757	0.092739	0.016275	0.44557
Mixte avec 2 classes	0.1241	0.087554	0.015401	0.42164
linear regression	0.12437	0.087327	0.015467	0.42344
Full data set	0,13499	0.099144	0.018222	0.49888
Full data set cut in 2 classes	0.13763	0.099092	0.018942	0.51857
Full data set cut in 3 classes	0.13501	0.096144	0.018227	0.49899
Removed Variables	0,134	0,097173	0,017957	0,49161
Communautés supprimées	0.12747	0.092553	0.016248	0.44483
Mixed with 2 classes	0.12437	0.087327	0.015467	0.42344
PLS regression 1st	0.12438	0.08572	0.015472	0.42357
Full data set	0,13347	0.09774	0.017815	0.48772
Full data set cut in 2 classes	0.13245	0.094019	0.017542	0.48025
Full data set cut in 3 classes	0.13047	0.091678	0.017021	0.466
Removed Variables	0,13291	0,09554	0,017665	0,48362
Communautés supprimées	0.12764	0.091114	0.016292	0.44602
Mixed with 2 classes	0.12438	0.08572	0.015472	0.42357
PLS regression advanced	0.1207	0.085773	0.01457	0.39888
Full data set	0.12743	0.093526	0.016238	0.44455
Full data set cut in 2 classes	0.12396	0.089755	0.015366	0.42067
Full data set cut in 3 classes	0.12021	0.087285	0.014451	0.39562
Removed Variables	0.12829	0.094293	0.016458	0.45057
Communautés supprimées	0.12429	0.088444	0.015448	0.4229
Mixed with 2 classes	0.1207	0.085773	0.01457	0.39888
SVM Polynomial	0.12175	0.08589	0.014822	0.40579
Full data set	0.12985	0.092377	0.01686	0.46268
Full data set cut in 2 classes	0.12911	0.088887	0.01667	0.45637
Full data set cut in 3 classes	0.13302	0.092129	0.017695	0.48444
Removed Variables	0,12925	0,089951	0,016705	0,45733
Communautés supprimées	0.12797	0.090735	0,017175	0,47019
Mixed with 2 classes	0.12175	0.08589	0.014822	0.40579
Neural network	0,11787	0,086258	0,013893	0,40909
Full data set	0,11787	0.086258	0.013893	0.40909
Full data set cut in 2 classes	0,13692	0.10066	0.018747	0.51323
Full data set cut in 3 classes	0.13393	0.094034	0.017938	0.4911
Removed Variables	0,13351	0,095503	0,017824	0,48797
Communautés supprimées	0.13552	0.094944	0.018367	0.50283
Mixed with 2 classes	0,13283	0,097711	0,017645	0,48306

Predict by removing extra community

2012-11-19T13:39:00.001+01:00

To ameliorate the estimation we can remove some non necessary values . The outliers can be removed by a statistical method.

In my case I use the cook distance . After a threshold distance we remove the communities.

Why by a distance ? The reason is: in our previous analysis we have found that our data are centered.

My conclusion is the distance permit to extract extrema values.

	RMSE	MAE	MSE	ARV
Régression linear	0.12747	0.092553	0.016248	0.44483
Régression PLS	0.12764	0.091114	0.016292	0.44602
SVM Polynomial	0.12797	0.090735	0,017175	0,47019
Neural network	0.13552	0.094944	0.018367	0.50283

Predict by removing unnecessary variables

2012-11-19T13:37:00.001+01:00

Some variables haven t sufficient effectiveness to give an important result. We can say that these values perturbate the model.

I have choosed two methods to remove some unnecessary variables .

- Low correlations (abs(corrélation)<0,1) : list 1 :householdsize, racePctAsian, agePct12t21, agePct16t24, agePct65up, pctUrban, pctWRetire, indianPerCap, PctEmplManu, PctEmplProfServ, PctWorkMomYoungKids, PersPerOccupHous, PctVacMore6Mos, MedOwnCostPctInc, MedOwnCostPctIncNoMtg, PctBornSameState, PctSameCity85, PctSameState85

- Opposite variables or our targetted values on the two by two matrix on the kohonen clustering research : list 2 : racePctWhite, pctUrban, pctWWage, pctWInvInc, pctWRetire, PctEmploy, PctFam2Par, PctKids2Par, PctYoungKids2Par, PctTeen2Par, PctWorkMom, PctSpeakEnglOnly, PersPerOccupHous, PersPerOwnOccHous, PctHousOccup, PctHousOwnOcc,MedYrHousBuilt, PctBornSameState, PctSameHouse85, PctSameCity85, PctSameState85

Removed values in two lists

	List	RMSE	MAE	MSE	ARV
linear regression	1	0,13825	0,10049	0,019112	0,52324
	2	0,134	0,097173	0,017957	0,49161
PLS regression	1	0,13671	0,098719	0,018691	0,5117
	2	0,13291	0,09554	0,017665	0,48362
SVM Polynomial	1	0,13048	0,091957	0,017025	0,46611
	2	0,12925	0,089951	0,016705	0,45733
Neural network	1	0,13788	0,09764	0,019011	0,52048
	2	0,13351	0,095503	0,017824	0,48797

Predict by classes

2012-11-19T13:33:00.000+01:00

In cutting our individuals data in more than one group , we can obtain more than one model. In using the SVM classification to automatically classify new data we obtain this result.

The result is better on all result. The best fit is for two or three classes. After this threshold the model become more unstable.

Portioning of ours classes

	Classe 1	Classe 2	Classe 3	Classe 4
4 classes	110	441	351	192
3 classes	381	596	117
2 classes	626	468

	Nb classes	RMSE	MAE	MSE	ARV
Linear regression	2	0.13763	0.099092	0.018942	0.51857
	3	0.13501	0.096144	0.018227	0.49899
	4	0.21412	0.13089	0.045847	1.2551
PLS regression	2	0.13245	0.094019	0.017542	0.48025
	3	0.13047	0.091678	0.017021	0.466
	4	0.14783	0.10862	0.021853	0.59828
SVM Polynomial	2	0.12929	0.092427	0.016715	0.4576
	3	0.12763	0.09117	0.016288	0.44593
	4	0.14822	0.1032	0.021969	0.60146
Neural network	2	0.13692	0.10066	0.018747	0.51323
	3	0.17428	0.13354	0.030374	0.83156
	4	0.18433	0.13045	0.033976	0.93016

Predict - Likelihood

2012-11-19T13:28:00.003+01:00

I introduce a semi parametric estimation . This method is really important. It s a statistical approach that not permit to have model but an estimation of this model.

We know two things in our analysis:

the first is the probability of the interconnection of Y (in our case the violent crimes ) and X (all our variables).
the second is the probability to know of all of our variables

The conditionnal probability of knowing the violent crimes (Y) giving our 99 variables(X) is:

The result estimation is

RMSE= 0.13475

Predict - Perceptron - Neural Network

2012-11-19T13:26:00.001+01:00

The percepton or the neural network approach is really an interesting method. Based on some method that use learning the time to obtain a good model is more long than other linear or mathematical study.

For verify the model evolution we use a validation period.

This is the result obtained with the neural network solution.

RMSE=0.11787
The overlearning permit to detect som instability during the learning . The best result is obtained with 40 round.

Compare round and error in learning and validation.

Compare three neurons size on learning errors.

Predict - Support Vector Machine

2012-11-19T13:24:00.003+01:00

The Support Vector Machine is based on the algorithm is given in wikipedia http://en.wikipedia.org/wiki/Support_vector_machine. I use the toolbox SVM KM.

I use a polynomial approach that offer an approaching quality as the linear approach.

RMSE=0.13

Predict - Full PLS regression advanced algo

2012-11-19T13:23:00.004+01:00

I use the toolbox glm-ie for a better result on the estimation.

The result is really better.

RMSE=0.12677

99 regression values

Variable	PLS value
'population'	0.3999
'householdsize'	0.0214
'racepctblack'	0.3249
'racePctWhite'	-0.0197
'racePctAsian'	-0.0128
'racePctHisp'	0.1544
'agePct12t21'	0.2107
'agePct12t29'	-0.2984
'agePct16t24'	-0.2222
'agePct65up'	0.0728
'numbUrban'	-0.6007
'pctUrban'	0.0616
'medIncome'	-0.0318
'pctWWage'	-0.027
'pctWFarmSelf'	0.0758
'pctWInvInc'	-0.202
'pctWSocSec'	0.2436
'pctWPubAsst'	0.053
'pctWRetire'	-0.1096
'medFamInc'	0.287
'perCapInc'	0.1205
'whitePerCap'	-0.3385
'blackPerCap'	-0.0349
'indianPerCap'	-0.0365
'AsianPerCap'	0.0113
'HispPerCap'	0.0438
'NumUnderPov'	0.0496
'PctPopUnderPov'	-0.167
'PctLess9thGrade'	-0.1638
'PctNotHSGrad'	0.0938
'PctBSorMore'	0.0608
'PctUnemployed'	-0.0264
'PctEmploy'	0.2514
'PctEmplManu'	-0.0542
'PctEmplProfServ'	0.0028
'PctOccupManu'	0.0836
'PctOccupMgmtProf'	0.0769
'MalePctDivorce'	0.3493
'MalePctNevMarr'	0.3719
'FemalePctDiv'	-0.0662
'TotalPctDiv'	-0.227
'PersPerFam'	-0.1277
'PctFam2Par'	-0.2185
'PctKids2Par'	0.0216
'PctYoungKids2Par'	-0.0836
'PctTeen2Par'	-0.0316
'PctWorkMomYoungKids'	0.075
'PctWorkMom'	-0.2358
'NumIlleg'	-0.0587
'PctIlleg'	-0.0566
'NumImmig'	-0.1606
'PctImmigRecent'	-0.0015
'PctImmigRec5'	0.0414
'PctImmigRec8'	0.0322
'PctImmigRec10'	-0.0792
'PctRecentImmig'	-0.1426
'PctRecImmig5'	0.153
'PctRecImmig8'	-0.126
'PctRecImmig10'	0.147
'PctSpeakEnglOnly'	0.1233
'PctNotSpeakEnglWell'	-0.1464
'PctLargHouseFam'	0.298
'PctLargHouseOccup'	-0.5543
'PersPerOccupHous'	0.5582
'PersPerOwnOccHous'	-0.0536
'PersPerRentOccHous'	-0.1228
'PctPersOwnOccup'	-0.5747
'PctPersDenseHous'	0.2394
'PctHousLess3BR'	0.1854
'MedNumBR'	0.0108
'HousVacant'	0.223
'PctHousOccup'	-0.0406
'PctHousOwnOcc'	0.4564
'PctVacantBoarded'	0.056
'PctVacMore6Mos'	-0.1005
'MedYrHousBuilt'	-0.0174
'PctHousNoPhone'	0.0764
'PctWOFullPlumb'	-0.0602
'OwnOccLowQuart'	-0.5692
'OwnOccMedVal'	0.6942
'OwnOccHiQuart'	-0.2407
'RentLowQ'	-0.2082
'RentMedian'	-0.1476
'RentHighQ'	-0.0501
'MedRent'	0.3854
'MedRentPctHousInc'	0.0655
'MedOwnCostPctInc'	0.0067
'MedOwnCostPctIncNoMtg'	-0.0985
'NumInShelters'	0.1403
'NumStreet'	0.1572
'PctForeignBorn'	0.203
'PctBornSameState'	0.0552
'PctSameHouse85'	0.0661
'PctSameCity85'	0.0252
'PctSameState85'	-0.0341
'LandArea'	0.1668
'PopDens'	0.0625
'PctUsePubTrans'	-0.0917
'LemasPctOfficDrugUn'	0.0052

Predict - Full PLS regression 1st algo

2012-11-19T13:22:00.001+01:00

The PLS approach is linear but more efficient. The description of this method is given in wikipedia:

http://en.wikipedia.org/wiki/Partial_least_squares_regression

The result is better than the classical linear regression approach.
RMSE=0.13347

Predict - Full Linear regression

2012-11-19T13:09:00.001+01:00

The first approach is purely linear. Based on the linear regression, we can identify some regressive values.

The first result is correct for this estimation.

99 regressive values

Variable	Regressive value
'population'	0.0037
'householdsize'	0.0043
'racepctblack'	0.2753
'racePctWhite'	-0.0102
'racePctAsian'	-0.0033
'racePctHisp'	0.0135
'agePct12t21'	0.0023
'agePct12t29'	-0.007
'agePct16t24'	-0.0033
'agePct65up'	0.0142
'numbUrban'	0.0036
'pctUrban'	0.0344
'medIncome'	0.0026
'pctWWage'	-0.0061
'pctWFarmSelf'	0.0072
'pctWInvInc'	-0.0124
'pctWSocSec'	0.0127
'pctWPubAsst'	0.0369
'pctWRetire'	-0.0105
'medFamInc'	0.0002
'perCapInc'	0.0027
'whitePerCap'	0.0032
'blackPerCap'	-0.0021
'indianPerCap'	-0.0028
'AsianPerCap'	0.0155
'HispPerCap'	0.006
'NumUnderPov'	0.005
'PctPopUnderPov'	0
'PctLess9thGrade'	-0.0025
'PctNotHSGrad'	0.0076
'PctBSorMore'	-0.0049
'PctUnemployed'	0.0003
'PctEmploy'	0.0028
'PctEmplManu'	-0.0042
'PctEmplProfServ'	-0.0013
'PctOccupManu'	0.0058
'PctOccupMgmtProf'	-0.0025
'MalePctDivorce'	0.123
'MalePctNevMarr'	0.0136
'FemalePctDiv'	0.0191
'TotalPctDiv'	0.0462
'PersPerFam'	0.008
'PctFam2Par'	-0.0207
'PctKids2Par'	-0.0355
'PctYoungKids2Par'	-0.0448
'PctTeen2Par'	-0.0131
'PctWorkMomYoungKids'	-0.0078
'PctWorkMom'	-0.0277
'NumIlleg'	0.0053
'PctIlleg'	0.1186
'NumImmig'	-0.0012
'PctImmigRecent'	-0.0027
'PctImmigRec5'	0.0004
'PctImmigRec8'	-0.0005
'PctImmigRec10'	0.0006
'PctRecentImmig'	0.0054
'PctRecImmig5'	0.0111
'PctRecImmig8'	0.0123
'PctRecImmig10'	0.0205
'PctSpeakEnglOnly'	0.0031
'PctNotSpeakEnglWell'	0.0038
'PctLargHouseFam'	0.0082
'PctLargHouseOccup'	0.0046
'PersPerOccupHous'	0.0083
'PersPerOwnOccHous'	-0.0001
'PersPerRentOccHous'	0.0136
'PctPersOwnOccup'	-0.0031
'PctPersDenseHous'	0.0809
'PctHousLess3BR'	0.0713
'MedNumBR'	-0.0067
'HousVacant'	0.1062
'PctHousOccup'	-0.0285
'PctHousOwnOcc'	0.0015
'PctVacantBoarded'	0.0292
'PctVacMore6Mos'	-0.0105
'MedYrHousBuilt'	0.0024
'PctHousNoPhone'	0.0181
'PctWOFullPlumb'	-0.0097
'OwnOccLowQuart'	0.0001
'OwnOccMedVal'	0.0034
'OwnOccHiQuart'	0.0033
'RentLowQ'	-0.0072
'RentMedian'	0.0033
'RentHighQ'	0.0072
'MedRent'	0.0077
'MedRentPctHousInc'	0.0389
'MedOwnCostPctInc'	0.0086
'MedOwnCostPctIncNoMtg'	-0.0336
'NumInShelters'	0.0126
'NumStreet'	0.0821
'PctForeignBorn'	0.0174
'PctBornSameState'	-0.0033
'PctSameHouse85'	0.0018
'PctSameCity85'	0.0357
'PctSameState85'	0.005
'LandArea'	0.0105
'PopDens'	0.0146
'PctUsePubTrans'	-0.0064
'LemasPctOfficDrugUn'	0.0094

RMSE=0.13499

The groups creation

2012-11-12T14:36:00.001+01:00

Summary

I group my last results on classification and clustering in this page and propose with a vote a last grouping organisation.

Variables on Group PCA/Visual Hierachical DBScan   kmeans SOM Vote Choice
agePct12t21 2   9   4 2 2
agePct12t29 2   9   4 2 2
agePct16t24 2 9 4 2 2
agePct65up 4 2 9 4 2 2
AsianPerCap 9 3 15 3 3 3
blackPerCap 10 3 15 3 3 3
FemalePctDiv 7 2 9 4 2 2
HispPerCap 9 3 12 3 3 3
householdsize 15 2 9 4 2 2
HousVacant 5 1 2 1 1
indianPerCap 3 2 2 1 2
LandArea 5 1 2 1 1
LemasPctOfficDrugUn 5 1 2 1 1
MalePctDivorce 7 2 9 4 2 2
MalePctNevMarr 2 9 4 2 2
medFamInc 10 3 12 3 3 3
medIncome 10 3 12 3 3 3
MedNumBR 2 3 2 3 2 3
MedOwnCostPctInc 6 2 2 4 3 2
MedOwnCostPctIncNoMtg 2 9 4 2 2
MedRent 3 7 3 3 3
MedRentPctHousInc 13 2 9 4 2 2
MedYrHousBuilt 2 9 4 4 2
numbUrban 5 1 2 1 1
NumIlleg 5 1 2 1 1
NumImmig 17 5 1 2 1 1
NumInShelters 5 1 2 1 1
NumStreet 5 1 2 1 1
NumUnderPov 5 1 2 1 1
OwnOccHiQuart 14 3 6 3 3 3
OwnOccLowQuart 14 3 6 3 3 3
OwnOccMedVal 14 3 6 3 3 3
PctBornSameState 4 4 8 5 4 4
PctBSorMore 10 3 12 3 2 3
PctEmplManu 5 2 9 4 2 2
PctEmploy 10 4 9 5 4 4
PctEmplProfServ 6 2 9 4 2 2
PctFam2Par 8 4 16 5 4 4
PctForreignBorn 18 5 3 1 1 1
PctHouseOccup 9 4 5 5 4 5
PctHouseOwnOccup 2 4 5 5 4 5
PctHousLess3BR 7 2 9 4 2 2
PctHousNoPhone 7 1 13 1 2 1
PctIlleg 7 1 10 1 1 1
PctImmigRec10 2 9 4 2 2
PctImmigRec5 2 9 4 2 2
PctImmigRec8 2 9 4 2 2
PctImmigRecent 2 9 4 2 2
PctKids2Par 8 4 16 5 4 4
PctLargHouseFam 5 4 1 1 1
PctLargHouseOccup 5 4 1 1 1
PctLess9thGrade 7 1 13 1 2 1
PctNotHSGrad 7 1 13 4 2 2
PctNotSpeakEnglWell 5 11 1 1 1
PctOccupManu 7 1 13 4 2 2
PctOccuptMgmtProf 10 3 12 3 2 3
PctPersDenseHous 5 11 1 1 1
PctPersOwnOccup 2 4 5 5 4 5
PctPopUnderPov 7 1 13 1 2 1
PctRecentImmig 18 5 3 1 1 1
PctRecImmig10 18 5 3 1 1 1
PctRecImmig5 18 5 3 1 1 1
PctRecImmig8 18 5 3 1 1 1
PctSameCity85 5 4 8 5 4 4
PctSameHouse85 3 2 14 5 4 4
PctSameState85 5 4 8 5 4 4
PctSpeakEnglOnly 1 4 2 5 4 4
PctTeen2Par 8 4 16 5 4 4
PctUnemployed 7 1 13 4 2 2
pctUrban 12 4 2 5 4 4
PctUsePubTrans 15 5 2 2 1 2
PctVacantBoarded 7 1 10 1 1 1
PctVacMore6Mo 4 2 9 4 2 2
pctWFarmSelf 6 3 2 4 2 2
pctWInvInc 8 4 2 5 4 4
PctWOFullPlumb 7 1 10 1 1 1
PctWorkMom 3 2 14 4 4 4
PctWorkMomYoungKids 5 2 14 4 4 4
pctWPubAsst 7 1 13 1 2 1
PctWRetire 3 2 14 4 4 4
pctWSocSec 4 2 9 4 2 2
pctWWage 11 4 9 5 4 4
PctYoungKids2Par 8 4 16 5 4 4
perCapInc 10 3 12 3 3 3
PersPerFam 17 2 9 4 2 2
PersPerOccupHous 15 2 9 4 2 2
PersPerOwnOccHous 15 2 9 4 4 2
PersPerRentOccHous 2 9 4 2 2
PopDens 17 5 4 1 1 1
population 5 1 2 1 1
racePctAsian 16 5 2 2 1 2
racepctblack 7 1 10 1 1 1
racePctHisp 5 11 1 1 1
racePctWhite 2 4 2 5 4 4
RenLowQ 13 3 7 3 3 3
RentHighQ 13 3 7 3 3 3
RentMedian 13 3 7 3 3 3
TotalPctDiv 7 2 9 4 2 2
ViolentCrimesPerPop 7 1 10 1 1 1
white per cap 10 3 12 3 3 3

Histogram

2012-11-09T14:18:00.000+01:00

Introduction

The values are centered and reduced. We need to analyse their distribution for a futur analysis based on a semi parametric statistic approach. On this case , I give all the distribution for all the variables.

Sometimes values are difficult to identify and two methods (mlc and em) are used to identify their distribution.

We can see by example that the more representative population of each community are based on low population number. We can see that the white people representation are more represented in this study. Black, asian,hispanic are lowly represented in the study.

We can see too than the urban people or illegitime children or not english speaking or immigrant or poverty are poorly represented. While their impact are really important in the violent crimes factor.

SOM on individuals with vote

2012-11-08T12:55:00.002+01:00

Introduction

Some of the individuals are difficult to identify but in exercising your eyes you should see a short code identifier of the state and the community name . With more than 1800 individuals , it s difficult to identify individual data. The method choose is the vote. The most representative of the area is represented on the view. More is white , more the group is large. More is blue , more it s desert.

The color is compared as an isle map. I use the sdh tools to generate this map.

SOM classification on variables

2012-11-08T12:53:00.004+01:00

Introduction

I use the SOM toolbox to extract my classification . It is a kohonen approach on Self Map Organisation.

It is a puissant tool to classify the metric data. Near a neural network method, the algorithm construct at each step a map reorganized to fit the best grouping on the selected map element (columns and rows).

This approach reveal than races are not an essential reference to classify the variables.

A Map of 6x6

This map is really well balanced.

On top left, we concentrate median people.

On top right, we concentrate nice people (white, speaking english ,...)

On bottom left , we concentrate immigrant and population density

On bottom right, we concentrate immigrant and social.

On bottom middle our value to predict the violent crime.

It is a good map. Each variable found a perfect place.

A Map of 2x2

I have reduced the map to a 2x2 to obtain more grouped variables. It s not innocent . It s because I need some opposable variables to our predict value for filter data in prediction research. You will see this feature in the next message.
Now on this map:
On the top left we have immigrant , illegitime,density, urban,poverty and our targeted variable to predict.
On the top right we have immigrant , manual employment,social,education.
On the bottom left we have medium family.
On the bottom right we found the ideal family speaking english with 2 children, living on the same area. The opposite of the violent crime

k-means clustering part 2

2012-10-25T13:27:00.001+02:00

Introduction

The must appropriate selection on variables is the cluster of five elements.
This grouping offers the particularity to give the same number of cluster than the agglomerative classification method.

The first group

PctForeignBorn,PctHousNoPhone,PctIlleg,PctLargHouseFam,PctLargHouseOccup,PctLess9thGrade,PctNotSpeakEnglWell,PctPersDenseHous,PctPopUnderPov,PctRecentImmig,PctRecImmig10,PctRecImmig5,PctRecImmig8,PctVacantBoarded,PctWOFullPlumb,pctWPubAsst,PopDens,racepctblack,racePctHisp,ViolentCrimesPerPop

The first group is composed by immigrant or illegitime children, living in large house not really graduate , not speaking english, under poverty, with public assistance, on a dense area. This group is near our targeted value.
Black and hispanic

The second group

HousVacant,indianPerCap,LandArea,LemasPctOfficDrugUn,numbUrban,NumIlleg,NumImmig,NumInShelters,NumStreet,NumUnderPov,PctUsePubTrans,population,racePctAsian
The group of workers whose use public transport under poverty with drug problems.
Asian and indian

The third group

AsianPerCap,blackPerCap,HispPerCap,medFamInc,medIncome,MedNumBR,MedRent,OwnOccHiQuart,OwnOccLowQuart,OwnOccMedVal,PctBSorMore,PctOccupMgmtProf,perCapInc,RentHighQ,RentLowQ,RentMedian,whitePerCap
The group of medium people from different race.

The fourth group

agePct12t21,agePct12t29,agePct16t24,agePct65up,FemalePctDiv,householdsize,MalePctDivorce,MalePctNevMarr,MedOwnCostPctInc,MedOwnCostPctIncNoMtg,MedRentPctHousInc,MedYrHousBuilt,PctEmplManu,PctEmplProfServ,PctHousLess3BR,PctImmigRec10,PctImmigRec5,PctImmigRec8,PctImmigRecent,PctNotHSGrad,PctOccupManu,PctUnemployed,PctVacMore6Mos,pctWFarmSelf,PctWorkMom,PctWorkMomYoungKids,pctWRetire,pctWSocSec,PersPerFam,PersPerOccupHous,PersPerOwnOccHous,PersPerRentOccHous,TotalPctDiv
The group of worker manual, immigrant , unemployed, with a big family , in a dense house and area.

The fifth group

PctBornSameState,PctEmploy,PctFam2Par,PctHousOccup,PctHousOwnOcc,PctKids2Par,PctPersOwnOccup,PctSameCity85,PctSameHouse85,PctSameState85,PctSpeakEnglOnly,PctTeen2Par,pctUrban,pctWInvInc,pctWWage,PctYoungKids2Par,racePctWhite
The perfect group , white and living on the same area, speaking in english with two kids, urban.
The white race disturb the idea of this group.