Epidemiology and Beyond

COVID-19 Related

2020-04-13T16:42:00.003-04:00

COVID-19 Related

Data Sources and Applications

Data repository for the 2019 Novel Coronavirus Visual Dashboard operated by the Johns Hopkins University
CDC: COVID-19 Case Surveillance Public Use Data with Geography
1Point3Acres
Apple: COVID-19 Mobility Trends Reports
C3.ai COVID-19 Data Lake is a unified, easily accessible data image of critical COVID-19 data publicly available at no cost to the global research community beginning on April 20, 2020.
SAS COVID-19 Resource Hub
reichlab/covid19-forecast-hub: Projections of COVID-19, in standardized format
CORD-19: COVID-19 Open Research Dataset
US Census: US Census Bureau COVID-19 Site
CDC: US Influenza Surveillance System, Laboratory-Confirmed Influenza Hospitalizations
European CDC: EARS-Net
How severe is the pandemic where you live?
新型コロナウイルス感染速報 - a shiny application (in Chinese) (English, Chinese)

Articles

CDC MMWR: COVID-19 related reports

SAS: Using data visualization to track the coronavirus outbreak
Stata by Chuck Huber

Brauer (2017): Mathematical epidemiology: Past, present, and future
Plus magazine: Covid-19 (SIR model)
software carpentry: Dynamic SIR Models using R! (deSolve)
Eugene Joh: SIR model with deSolve & ggplot2
Wikipedia: Compartmental models in epidemiology
Lancet: COVID-19 Resource Centre
NEJM: Coronavirus
HealthKnowledge: Epidemic theory (R0, R, Herd immunity, etc.)
Bjørnstad: R package for Epidemics: Models and Data in R
Frosts: Epi Recipes of Epi Cookbook

Video

3Blue1Brown: Exponential growth and epidemics; Taylor series

Prevalence of Diabetes by Race and Ethnicity in the United States, 2011-2016

2019-12-26T10:03:00.000-05:00

Prevalence of Diabetes by Race and Ethnicity in the United States, 2011-2016
YJ Cheng, AM Kanaya, MRG Araneta, et al. JAMA 2019;322(24):2389-2398

Key Points

Question During 2011-2016, how prevalent was diabetes among major race/ethnicity groups and subgroups of Hispanic and non-Hispanic Asian adults in the United States?
Findings In this cross-sectional study that included 7575 adults, the age- and sex-adjusted diabetes prevalence was 12.1% for non-Hispanic white, 20.4% for non-Hispanic black, 22.1% for Hispanic, and 19.1% for non-Hispanic Asian groups. The diabetes prevalence also differed significantly among Hispanic or non-Hispanic Asian subgroups.
Meaning In the United States in 2011-2016, the prevalence of diabetes varied across racial/ethnic groups.
...Full text...

News etc.

Editorial: K Matsushita, O Tang, E Selvin. Addressing Challenges and Implications of National Surveillance for Racial/Ethnic Disparities in Diabetes
CDC Newsroom: CDC Releases First National Estimates on Diabetes within Hispanic and Asian Populations in the US
U.S. News: CDC Study Breaks Down Diabetes Risk for Hispanic, Asian Subgroups
abs NEWS: First national estimates on how common diabetes is among Hispanic and Asian Americans: CDC
DailyMail: Diabetes is TWICE as common in Asian and Hispanic Americans as the disease is among the US population, CDC says
JAMA Podcast: Diabetes Prevalence in the US, Invasive Nontypeable ..., and more
Medicine Matters Diabetes: Diabetes prevalence varies between and within US ethnic subgroups
Altmetric

A brief history of AI, machine learning, artificial neural networks, and deep learning

2019-05-17T10:04:00.003-04:00

A brief history of AI, machine learning, artificial neural networks, and deep learning
TheScientist, Jef Akst (2019)
"The term “artificial intelligence” dates back to the mid-1950s, when mathematician John McCarthy, widely recognized as the father of AI, used it to describe machines that do things people might call intelligent. He and Marvin Minsky, whose work was just as influential in the AI field, organized the Dartmouth Summer Research Project on Artificial Intelligence in 1956. A few years later, with McCarthy on the faculty, MIT founded its Artificial Intelligence Project, later the AI Lab. It merged with the Laboratory for Computer Science (LCS) in 2003 and was renamed the Computer Science and Artificial Intelligence Laboratory, or CSAIL"

effect size, P value, and Bayes odds

2019-05-15T10:00:00.004-04:00

P value, effect size, and Bayes factor, and after one year

2021

Cole (2020): Surprise! (P value vs. S value)
KDnuggets by Bills (2021): Null Hypothesis Significance Testing is Still Useful

2019:Another wave beyond the P-value:

Greenland (2019): Valid p-values behave exactly as they should: Some misleading criticisms of p-values and their resolution with s-values.
Benjamin (2019): Three recommendations for improving the use of p-values
Blume (2019): An introduction to second-generation p-values
Tarran (2019): Is this the end of "statistical significance"?

American Statistician (2019): A few articles relate to the p-value and alternative
American Statistician (2019 Supl.): Statistical Inference in the 21st Century: A World Beyond p < 0.05

Wasserstein (2019): Moving to a World Beyond “p < 0.05”

McShane (2019): Abandon Statistical Significance

Amrhein (2019): Comment of Nature: Scientists rise up against statistical significance,
Editorial of Nature (2019): It’s time to talk about ditching statistical significance

Different opinions

Adams (2019): a trillion P values and counting
Zhang (2019): P values akin to ‘beyond reasonable doubt’

2017: Robert Matthews published an article about the changes of statistical practice after ASA's statement on the statistical significance and p-values: The ASA's p-value statement, one year on. I agree on one highlight in this article: "It should be possible to establish firm general principles which focus on what is right rather than what is wrong"

Abstract:Its aim was to stop the misuse of statistical significance testing. But Robert Matthews argues that little has changed in the 12 months since the ASA's intervention.
Why do people use p-values instead of computing probability of the model given data?

2016: ASA releases statement on statistical significance and p-values (03/07/2016). The statement's six principles:

P-values can indicate how incompatible the data are with a specified statistical model.
P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
Proper inference requires full reporting and transparency.
A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

p-value is the likelihood of null hypothesis (H0), a conditional probability given H0:

$p (X \geq x|H 0)$ for a right tail event
p(X ≤ x|H0) for a left tail event
2 * min{ $P (X \geq x|H 0), p (X \leq x | H 0)$ } for a 2-tail event
α level, the level of significance, is a pre-defined probability of falsely rejecting the null hypothesis we accept. Typically, the statistical significance means the p-value < α level at 0.05.

False discovery rate (Wikipedia)

Many times, the p-value was incorrectly interpreted as one of posterior probabilities given the observed data, probabilities given the observed data, the false discovery rate (Colquhoun, 2014) or false positive rate.
Storey (2003): The Positive False Discovery Rate: A Bayesian Interpretation and the q-Value

Type I error (α) = 1 - specificity, Type II error (β) = 1 - sensitivity, Power = sensitivity = 1 - β
Leek (2017). Five ways to fix statistics (Nature)
Benjamin (2017). Redefine statistical significance
Baker (2016). Statisticians issue warning over misuse of P values (pdf)
Matloff (2016). 1) After 150 years, the ASA says no to p-value, 2) Further comments on the ASA manifesto, 3) P-values: the continuing saga
Benjamini & Galili (2016). It's not the p-values' fault reflections on the recent ASA statement
Kass (2016). Ten Simple Rules for Effective Statistical Practice. It acknowledged three websites:

xkcd.com "for conveying statistical ideas with humor"
Simply Statistics "for thoughtful commentary"
FiveThirtyEight "for bringing statistics to the world (or at least to the media)".

Aschwanden (2016). Science Isn’t Broken - It’s just a hell of a lot harder than we give it credit for
Halsey (2015). The fickle P value generates irreproducible results
Lazzeroni (2016). Solutions for quantifying P value uncertainty and replication power, and response of Halsey;
Nuzzo (2013). Scientific method: Statistical errors

Epimonitor (2016). Growing Concern About Statistical Errors Triggers Statement of P-Values

Youngquist (2012). Part 19: What is a P value;

GraphPad's Advice: how to interpret a small P value

Frost (2014). How to Correctly Interpret P Values, Five Guidelines for Using P values

Held (2010). A nomogram for P values
Cumming (2012). Mind your confidence interval: how statistics skew research results
Gumming (2013). The problem with p values: how significant are they, really?
Ioannidis (2015). Why most published research findings are false
Ranstam (2012). Why the P value culture is bad and confidence intervals a better alternative. Ranstam (2009) Sampling uncertainty in medical research. Austin (2002). A brief note on overlapping confidence intervals
Aschwanden. Statisticians found one thing they can agree on: it's time to stop misusing P-values.
Lew (2013). Give p a chance: significance testing is misunderstood
Sullivan (2012). Using Effect Size—or Why the P Value Is Not Enough.
Fraser (2016). Crisis in Science? or Crisis in Statistics! Mixed messages in Statistics with impact on Science
Gelman (2014). Data-dependent analysis—a “garden of forking paths”—
Capital of Statistics. 美国统计协会开始正式吐槽（错用）P值啦
Wikipedia. Type I & II errors, sensitivity & specificity, effect size, Bayes factor.
Hubers (2013). Measures of effect size in Stata 13
Robert Coe (2002). It's the Effect Size, Stupid
Andrew Gelman (2013). P value and statistical practice, Misunderstanding the p-value
Simonsohn (2013). Just Post It: The Lesson From Two Cases of Fabricated Data Detected by Statistics Alone
Goodman (2001). Of P-values and Bayes: a modest proposal
Goodman (1999). Toward evidence-based medical statistics: the P value fallacy and The Bayes factor (notes: I like the Kass's formula, which uses the likelihood of alternative hypothesis as a numerator, and gives a BF without many decimals).
Kass (1995). Bayes factors [on the basis of observed data D, for the dichotomous conditions / models / hypotheses (H1 or H0), Bayes factor = p(D|H1)/p(D|H0)], the rules of thumb assess the quality of the evidence favoring one hypothesis over another as a reference:

1 to 3 (not worth more than a bare mention)
3 to 20 (positive)
20 to 150 (strong)
> 150 (very strong)

The odds form of Bayes's theorem for two hypotheses is convenient for calculating a Bayesian update of a chance.

If there are mutually exclusive hypotheses H1 (= Alternative hypothesis, H1 = Disease) and H0 (= Null hypothesis = No disease) by given D (= Observed data/evidence = Postive test),

p(H1 and D) = p(H1|D)*p(D) = p(D|H1)*p(H1)
p(H1|D) = p(H1)*p(D|H1)/p(D)
p(H0|D) = p(H0)*p(D|H0)/p(D)
p(H1|D)/p(H0|D) = p(H1)/p(H0)*p(D|H1)/p(D|H0)
p(H1|D)/[1-p(H1|D)] = p(H1)/[1-p(H1)]*p(D|H1)/p(D|H0)
odds(H1|D) = odds(H1)*p(D|H1)/p(D|H0)

Bayes factor (BF) = p(D|H1)/p(D|H0)

Prior odds of H1 = odds(H1) = p(H1)/p(H0) = p(H1)/[1-p(H1)]
Posterior odds of H1 given data = odds(H1|D) = p(H1|D)/p(H0|D)= (prior odds)*(BF or likelihood ratio)
p(H1)=odds(H1)/[1+odds(H1)], p(H1|D)=odds(H1|D)/[1+odds(H1|D)]

Suppose a person with disease had 3/4 possiblity of positive test, and a person without disease had 1/5 possibility of positive test. and we have no idea of p(D) of that person (the population), which means the prior proability is 50% and the prior odds(D) = p(D) / (1 - p(D) = 1:1. When a person had a positive test:

the posterior odds (odds(D|Pos)) = (1:1) * (3/4 / 1/5) = 15/4 = 3.75 = (p(D) / (1-p(D)) * (p(Pos|D) / (1 - p(Pos|D)) = odds(D) * BF
or, the probability had a disease given a positive test (p(D|Pos)) = ((3/4) * 0.5) / (0.5 * 3/4 + (1 - 0.5) * 1/5) = 15/19 = odds(D|Pos) / (1 + odds(D|Pos)) = .79

Eight versions of Bayes's Theorem (pdf): simple, explicity, general, Sigma, canceled, odds, relative odds, and compound odds.

Currell (2009). Chapter 7 Bayesian statistics

O’Hagan (2006). Bayes factors, Deeks (2004). Diagnostic tests 4: likelihood ratios, Lindley (2004). Bayesian thoughts, Griffis (2006). Statistics and the Bayesian mind
Berger (1988). The likelihood principle: a review, generalizations, and statistical implications
Masson (2011): A tutorial on a practical Bayesian alternative to null-hypothesis significance testing
Faulkenberry (2018): A Simple Method for Teaching Bayesian Hypothesis Testing in the Brain and Behavioral Sciences
Sellke (2001): Calibration of p Values for Testing Precise Null Hypotheses
Berger (1987): Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Bayesian p-value

Nowozin (2015). Bayesian P-Values
Lin (2009). Using Bayesian p-values in a 2 x 2 table of matched pairs with incompletely classified data

NISS Webinar (2019): Alternatives to the Traditional P-value

Choosing R or Python for Data Analysis? An Infographic

2018-12-17T10:39:00.000-05:00

Choosing R or Python for Data Analysis? An Infographic
by Karlijn Willems

Wondering whether you should use R or Python for your next data analysis post? Check our infographic "Data Science Wars: R vs Python".
I think you'll agree with me if I say:
It's HARD to know whether to use Python or R for data analysis. And this is especially true if you're a newbie data analyst looking for the right language to start with.
It turns out that there are many good resources that can help you to figure out the strengths and weaknesses of both languages. They often go into great detail, and provide a tailored answer to questions such as "What should I use for Machine Learning?", or "I need a fast solution, should I go for Python or R?".
In today's post, I present to you our new Infographic "Data Science Wars: R vs Python", that highlights in great detail the differences betweens these two languages from a data science point of view. So that next time you are debating R vs Python for machine learning, statistics, or maybe even the Internet of Things, you can have a look at the infographic and find the answer. ...

R or Python? Why not both? Using Anaconda Python within R with {reticulate}

Stata undocumented commands and options

2018-10-04T12:50:00.001-04:00

Stata undocumented commands and options
Stata undocumented/obsolete commands and options are usually used by limited people such as Stata programmers of StataCorp in developing Stata. However, sometimes I find they are quite handy.

Analyzing survey data: -svygen- (generating adjusted sampling weights), -svycal- (calibration-adjusted sampling weights)
Generating the estimate variables of a graph: - twoway__function_gen-, -twoway__histogram_gen-, -twoway__kdensity_gen-
Calculating margins: -margins, gen()- (create margin-response variables in the current dataset), -margins, saving()- (save margins results to a dataset)
Programming: -set buildfvinfo- (build extra factor-variables information into parameter estimates), -_coef_table- (displaying estimation results)
finding more: A list of Stata undocumented command
-vce()- options and options for survey data analysis: -vce(svy)- and -vce(unconditional)- are obsolete options of -vce(linearized)-

verification or validation

2018-10-04T11:54:00.001-04:00

Verification or Validation

Verification and validation are quite different and used for different purposes. For example, verification answers, "Am I building the model right?", and validation answers, "Am I building the right model?"
Wikipedia: Verification and validation of computer simulation models
Wikipedia: Verification and validation
Verification: The evaluation of whether or not a product, service, or system complies with a regulation, requirement, specification, or imposed condition. It is often an internal process
Validation: The assurance that a product, service, or system meets the needs of the customer and other identified stakeholders. It often involves acceptance and suitability with external customers.

Stata: output and reuse the estimates of histogram

2018-10-02T17:45:00.001-04:00

Stata: Output and reuse the estimates of a histogram
Sometimes we would like to export the estimates of a histogram and make a histogram plot using other software such as Excel. The undocumented command - twoway__ histogram_gen - can help.

Stata tip 20: Generating histogram bin variables, for example (modified):

sysuse auto,clear
histogram weight, percent bin(20)
twoway__histogram_gen weight, bin(20) percent gen (hvalue xvalue, replace)
twoway bar hvalue xvalue, barwidth(.15) bstyle(histogram)
list hvalue xvalue if !missing(hvalue), sum(hvalue) clean
or, you can copy columns from the "Data Editor" window and paste them into Excel sheet.

or use - serset -, the data in the .gph file are stored in the serset format

sysuse auto,clear
histogram weight, percent width(100)
preserve
serset use,clear
then, you can copy the first and third columns (such as __000001, __000003) from the "Data Editor" window and paste them into an Excel sheet.
restore

- grexport - a user-written ado file, an alternative to -graph export-

sysuse auto,clear
histogram weight, percent width(100)
grexport, list //list results in the result window

Curve-Fitting

2018-09-29T15:18:00.001-04:00

XKCD: Curve-Fitting

Stata: Output the results into text files

2018-07-14T11:37:00.001-04:00

Stata: Output the results into text files

Stata manual

UCLA: How do I use to write results or other information to an external text file?
Example:

cap program drop coeftab
program coeftab
args per dec wth
#del ;
if "`per'"=="" {; local per=100; };
if "`dec'"=="" {; local dec=2; };
if "`wth'"=="" {; local wth=4; };
#del cr
mat R=r(table)'
mat rtable=R[1...,"b"]
local rows=rowsof(rtable)
local names: rownames rtable
disp _skip(24) "estimate (LL, UL) estimate (SE) p-value"
forvalues i=1/`rows' {
disp "`:word `i' of `names''" _column(25) ///
%`wth'.`dec'f rtable[`i',1] ///
" (" %`wth'.`dec'f rtable[`i',2] ///
", " %`wth'.`dec'f rtable[`i',3] ")" ///
_skip(10) %`wth'.`dec'f rtable[`i',1] ///
" (" %`wth'.`dec'f rtable[`i',4] ///
")" _skip(10) %5.3f rtable[`i',5]
}
end

Stata: Output the results into Excel tables

2018-07-14T10:02:00.003-04:00

Stata: Output the results into Excel tables

Stata manual: putexcel, putexcel advanced
Stata blogs: Chuck Huber (2017).Creating Excel tables with putexcel

ASCII Codes

Example:
cd c:\temp sysuse auto regress price mpg i.foreign margins foreign mat rtable=r(table)' putexcel set demo.xlsx,sheet(demo) replace local cell1=char(64+1)+string(1) local cell2=char(64+1)+string(5) putexcel `cell1'=matrix(rtable) /// `cell2'=matrix(rtable[1...,1],rtable[1...,5..6],rtable[1...,2]),names

The Great Big Data Science Glossary

2018-02-28T09:47:00.004-05:00

The Great Big Data Science Glossary
by Wolf Howard
Getting started in data science can be overwhelming, especially when you consider the variety of concepts and techniques a data scienctist needs to master in order to do her job effectively. Even the term "data science" can be somewhat nebulous, and as the field gains popularity it seems to lose definition.

To help those new to the field stay on top of industry jargon and terminology, we've put together this glossary of data science terms. We hope it will serve as your handy quick reference whenever you're working on a project, or reading an article and find you can't quite remember what "ETL" means.

Fundamentals

These are some baseline concepts that are helpful to grasp when getting started in data science. While you probably won't have to work with every concept mentioned here, knowing what the terms mean will help when reading articles or discussing topics with fellow data lovers.
Algorithms
An algorithm is a set of instructions we give a computer so it can take values and manipulate them into a usable form. This can be as easy as finding and removing every comma in a paragraph, or as complex as building an equation that predicts how many home runs a baseball player will hit in 2018.
Back End
The back end is all of the code and technology that works behind the scenes to populate the front end with useful information. This includes databases, servers, authentication procedures, and much more. You can think of the back end as the frame, the plumbing, and the wiring of an apartment.
Big Data
Big data is a term that suffers from being too broad to be useful. It’s more helpful to read it as, “so much data that you need to take careful steps to avoid week-long script runtimes.” Big data is more about strategies and tools that help computers do complex analysis of very large (read: 1+ TB) data sets. The problems we must address with big data are categorized by the 4 V's: volume, variety, veracity, and velocity.
Classification
Classification is a supervised machine learning problem. It deals with categorizing a data point based on its similarity to other data points. You take a set of data where every item already has a category and look at common traits between each item. You then use those common traits as a guide for what category the new item might have.
Database
As simply as possible, this is a storage space for data. We mostly use databases with a Database Management System (DBMS), like PostgreSQL or MySQL. These are computer applications that allow us to interact with a database to collect and analyze the information inside.
Data Warehouse
A data warehouse is a system used to do quick analysis of business trends using data from many sources. They're designed to make it easy for people to answer important statistical questions without a Ph.D. in database architecture.
Front End
The front end is everything a client or user gets to see and interact with directly. This includes data dashboards, web pages, and forms.
Fuzzy Algorithms
Algorithms that use fuzzy logic to decrease the runtime of a script. Fuzzy algorithms tend to be less precise than those that use Boolean logic. They also tend to be faster, and computational speed sometimes outweighs the loss in precision.
Fuzzy Logic
An abstraction of Boolean logic that substitutes the usual True and False and for a range of values between 0 and 1. That is, fuzzy logic allows statements like "a little true" or "mostly false."
Greedy Algorithms
A greedy algorithm will break a problem down into a series of steps. It will then look for the best possible solution at each step, aiming to find the best overall solution available. A good example is Dijkstra's algorithm, which looks for the shortest possible path in a graph.
Machine Learning
A process where a computer uses an algorithm to gain understanding about a set of data, then makes predictions based on its understanding. There are many types of machine learning techniques; most are classified as either supervised or unsupervised techniques.
Overfitting
Overfitting happens when a model considers too much information. It’s like asking a person to read a sentence while looking at a page through a microscope. The patterns that enable understanding get lost in the noise.
Regression
Regression is another supervised machine learning problem. It focuses on how a target value changes as other values within a data set change. Regression problems generally deal with continuous variables, like how square footage and location affect the price of a house.
Statistic vs. Statistics
Statistics (plural) is the entire set of tools and methods used to analyze a set of data. A statistic (singular) is a value that we calculate or infer from data. We get the median (a statistic) of a set of numbers by using techniques from the field of statistics.
Training and Testing
This is part of the machine learning workflow. When making a predictive model, you first offer it a set of training data so it can build understanding. Then you pass the model a test set, where it applies its understanding and tries to predict a target value.
Underfitting
Underfitting happens when you don’t offer a model enough information. An example of underfitting would be asking someone to graph the change in temperature over a day and only giving them the high and low. Instead of the smooth curve one might expect, you only have enough information to draw a straight line.

Fields of Focus

As businesses become more data-focused, new opportunities open up for people of various skill sets to become part of the data community. These are some of the areas of specialization that exist within the data science realm.
Artificial Intelligence (AI)
A discipline involving research and development of machines that are aware of their surroundings. Most work in A.I. centers on using machine awareness to solve problems or accomplish some task. In case you didn’t know, A.I. is already here: think self-driving cars, robot surgeons, and the bad guys in your favorite video game.
Business Intelligence (BI)
Similar to data analysis, but more narrowly focused on business metrics. The technical side of BI involves learning how to effectively use software to generate reports and find important trends. It’s descriptive, rather than predictive.
Data Analysis
This discipline is the little brother of data science. Data analysis is focused more on answering questions about the present and the past. It uses less complex statistics and generally tries to identify patterns that can improve an organization.
Data Engineering
Data engineering is all about the back end. These are the people that build systems to make it easy for data scientists to do their analysis. In smaller teams, a data scientist may also be a data engineer. In larger groups, engineers are able to focus solely on speeding up analysis and keeping a data well organized and easy to access.
Data Journalism
This discipline is all about telling interesting and important stories with a data focused approach. It has come about naturally with more information becoming available as data. A story may be about the data or informed by data. There’s a full handbook if you’d like to learn more.
Data Science
Given the rapid expansion of the field, the definition of data science can be hard to nail down. Basically, it’s the discipline of using data and advanced statistics to make predictions. Data science is also focused on creating understanding among messy and disparate data. The “what” a scientist is tackling will differ greatly by employer.
Data Visualization
The art of communicating meaningful data visually. This can involve infographics, traditional plots, or even full data dashboards. Nicholas Felton is a pioneer in this field, and Edward Tufte literally wrote the book.
Quantitiative Analysis:
This field is highly focused on using alogrithms for to gain an edge in the financial sector. These algorithms either recommend or make trading decisions based on a huge amount of data, often on the order of picoseconds. Quantitative analysts are often called "quants."

Statistical Tools

There are a number of statistics data professionals use to reason and communicate information about their data. These are some of the most basic and vital statistical tools to help you get started.
Correlation
Correlation is the measure of how much one set of values depends on another. If values increase together, they are positively correlated. If one values from one set increase as the other decreases, they are negatively correlated. There is no correlation when a change in one set has nothing to do with a change in the other.
Mean (Average, Expected Value)
A calculation that gives us a sense of a "typical" value for a group of numbers. The mean is the sum of a list of values divided by the number of values in that list. It can be deceiving used on its own, and in practice we use the mean with other statistical values to gain intuition about our data.
Median
In a set of values listed in order, the median is whatever value is in the middle. We often use the median along with the mean to judge if there are values that are unusually high or low in the set. This is an early hint to explore outliers.
Normalize
A set of data is said to be normalized when all of the values have been adjusted to fall within a common range. We normalize data sets to make comparisons easier and more meaningful. For instance, taking movie ratings from a bunch of different websites and adjusting them so they all fall on a scale of 0 to 100.
Outlier
An outlier is a data point that is considered extremely far from other points. They are generally the result of exceptional cases or errors in measurement, and should always be investigated early in a data analysis workflow.
Sample
The sample is the collection of data points we have access to. We use the sample to make inferences about a larger population. For instance, a political poll takes a sample of 1,000 Greek citizens to infer the opinions of all of Greece.
Standard Deviation
The standard deviation of a set of values helps us understand how spread out those values are. This statistic is more useful than the variance because it's expressed in the same units as the values themselves. Mathematically, the standard deviation is the square root of the variance of a set. It's often represented by the greek symbol sigma, σ.
Statistical Significance
A result is stasticially significant when we judge that it probably didn't happen due to chance. It is highly used in surveys and statistical studies, though not always an indication of pratical value. The mathematical details of statistical significance are beyond the scope of this post, but a fuller explanation can be found here.
Summary Statistics
Summary statistics are the measures we use to communicate insights about our data in a simple way. Examples of summary statistics are the mean, median and standard deviation.
Time Series
A time series is a set of data that's ordered by when each data point ocurred. Think of stock market prices over the course of a month, or the temperature throughout a day.
Residual (Error)
The residual is a measure of how much a real value differs from some stastical value we calculated based on the set of data. So given a prediction that it will be 20 degrees fahrenheit at noon tomorrow, when noon hits and its only 18 degrees, we have an error of 2 degrees. This is often used interchangably with the term "error," even though, technically, error is a purely theoretical value.
Variance
The variance of a set of values measures how spread out those values are. Mathematically, it is the average difference between individual values and the mean for the set of values. The square root of the variance for a set gives us the standard deviation, which is more intuitively useful.

Parts of a Workflow

While every workflow is different, these are some of the general processes that data professionals use to derive insights from data.
Data Exploration
The part of the data science process where a scientist will ask basic questions that helps her understand the context of a data set. What you learn during the exploration phase will guide more in-depth analysis later. Further, it helps you recognize when a result might be surprising and warrant further investigation.
Data Mining
The process of pulling actionable insight out of a set of data and putting it to good use. This includes everything from cleaning and organizing the data; to analyzing it to find meaningful patterns and connections; to communicating those connections in a way that helps decision-makers improve their product or organization.
Data Pipelines
A collection of scripts or functions that pass data along in a series. The output of the first method becomes the input of the second. This continues until the data is appropriately cleaned and transformed for whatever task a team is working on.
Data Wrangling (Munging)
The process of taking data in its original form and "taming" it until it works better in a broader workflow or project. Taming means making values consistent with a larger data set, replacing or removing values that might affect analysis or performance later, etc. Wrangling and munging are used interchangeably.
ETL (Extract, Transform, Load)
This process is key to data warehouses. It describes the three stages of bringing data from numerous places in a raw form to a screen, ready for analysis. ETL systems are generally gifted to us by data engineers and run behind the scenes.
Web Scraping
Web scraping is the process of pulling data from a website’s source code. It generally involves writing a script that will identify the information a user wants and pull it into a new file for later analysis.

Machine Learning Techniques

The field of machine learning has grown so large that there are now positions for Machine Learning Engineers. The terms below offer a broad overview of some common techniques used in machine learning.
Clustering
Clustering techniques attempt to collect and categorize sets of points into groups that are “sufficiently similar," or "close" to one another. "Close" varies depending on how you choose to measure distance. Complexity increases as the more features are added to a problem space.
Decision Trees
This machine learning method uses a line of branching questions or observations about a given data set to predict a target value. They tend to over-fit models as data sets grow large. Random forests are a type of decision tree algorithm designed to reduce over-fitting.
Deep Learning
Deep learning models use very large neural networks — called deep nets — to solve complex problems, such as facial recognition. The layers in a model start with identifying very simple patterns and then build in complexity. By the end the net (hopefully) has a nuanced understanding that can accurately classify or predict values.
Feature Engineering
The process of taking knowledge we have as humans and translating it into a quantitative value that a computer can understand. For example, we can translate our visual understanding of the image of a mug into a representation of pixel intensities.
Feature Selection
The process of identifying what traits of a data set are going to be the most valuable when building a model. It's especially helpful with large data sets, as using fewer features will decrease the amount of time and complexity involved in training and testing a model. The process begins with measuring how relevant each feature in a data set is for predicting your target variable. You then choose a subset of features that will lead to a high-performance model.
Neural Networks
A machine learning method that’s very loosely based on neural connections in the brain. Neural networks are a system of connected nodes that are segmented into layers — input, output, and hidden layers. The hidden layers (there can be many) are the heavy lifters used to make predictions. Values from one layer are filtered by the connections to the next layer, until the final set of outputs is given and a prediction is made. A nice video explanation can be found here.
Supervised Machine Learning
With supervised learning techniques, the data scientist gives the computer a well-defined set of data. All of the columns are labelled and the computer knows exactly what it’s looking for. It’s similar to a professor handing you a syllabus and telling you what to expect on the final.
Unsupervised Machine Learning
In unsupervised learning techniques, the computer builds its own understanding of a set of unlabeled data. Unsupervised ML techniques look for patterns within data, and often deal with classifying items based on shared traits.

SAS: LSMESTIMATE vs CONTRAST and ESTIMATE

2018-02-27T16:01:00.000-05:00

SAS: LSMESTIMATE vs CONTRAST and ESTIMATE
LSMESTIMATE was a new statement in SAS 9.22 and much easier to be used than CONTRAST and ESTIMATE

Wikipedia.Contrast (Statistics)
Kiernan (2011). CONTRAST and ESTIMATE Statements Made Easy: The LSMESTIMATE Statement
Wicklin (2016). How to write CONTRAST and ESTIMATE statements in SAS regression procedures
A quick guide to writing contrast statements in SAS and SPSS,here is a Chinese version

SAS: how to customize the plots

2018-02-27T15:07:00.001-05:00

SAS: How to Customize the Plots

Usually I prefer to get a dataset of estimates for the plot, then use PROC SGPLOT/GPLOT, R!, or MS Excel plot those estimates.
Or, modify the template of a statistic plot following the steps:

Turn on the trace function: "ODS TRACE ON;"; run the PROC XXX; then find the name of default template (such as 'Stat.Lifetest.Graphics.cifPlot'); then "ODS TRACE OFF;"
Print out the template: "PROC TEMPLATE; SOURCE Stat.Lifetest.Graphics.cifPlot; RUN;"
Copy the definition from the log window
Paste it in underscore part: "PROC TEMPLATE; _________; RUN;" and modify the definition part you are interested in, and run it
Run the PROC XXX again.
After done: run "PROC TEMPLATE; DELETE Stat.Lifetest.Graphics.cifPlot; RUN;" to set the template as the default format.

SAS/STAT Document 9.3, 14.3:

May use the "Tools -> ODS Graphics Editor" to edit these plots after you get them. It's better use SGE format by setting "ODS LISTING SGE=ON; ODS HTML SGE=ON;"
Slaughter (2010).Using PROC SGPLOT for Quick High-Quality Graphs
Amruthnath (2013). PROC SGPLOT over PROC GPLOT
Kuhfeld (2013). Creating and customizing the Kaplan-Meier survival plot in PROC LIFETEST large graphics
Zender (2011). The Greatest Hits: ODS Essentials Every User Should Know
Zender (2009). Tiptoe through the Templates
Smith (2007). The Output Delivery System (ODS) from Scratch
Wicklin (2012): How to get data values out of ODS graphics

    ODS GRAPHICS ON;
    ODS TRACE ON;
    ODS OUTPUT Histogram=HIST;
      PROC UNIVARIATE DATA=SAShelp.cars NOPRINT;
      VAR invoice;
      HISTOGRAM invoice;
    RUN;
    ODS TRACE OFF;

SAS: Change folder directory

2018-02-27T09:20:00.001-05:00

SAS: How to Change Folder Directory

UCLA: How do I locate the SAS temporary work directory?
Check and change current folder:

check: %put === Current path is: %sysfunc(pathname(&fr)) ===;
double-click the dialog box at right bottom corner, then change it to the folder you prefer
use SAS command 'X' in a SAS program: X 'cd "c:\XYZ"';

Set SAS work directory:

Effect of population screening for type 2 diabetes

2017-08-24T11:15:00.000-04:00

Effect of population screening for type 2 diabetes
by Medical Xpress August 24, 2017
"Three large trials published today in Diabetologia (the journal of the European Association for the Study of Diabetes) show that screening for type 2 diabetes and cardiovascular risk factors may not reduce mortality and cardiovascular disease in the general population. However, for individuals diagnosed with diabetes, screening is associated with a reduction in mortality and cardiovascular disease risk. " full text

Articles mentioned in this news:

continuing professional education

2017-08-11T10:45:00.001-04:00

Continuing Professional Education & Glossary by JECH

2008

Hruschka (2008/11). A glossary of culture in epidemiology
Gunasekara (2008/10). Glossary for econometrics and epidemiology
Freeman (2008/9).Gone viral? Heard the buzz? A guide for public health practitioners and researchers on how Web 2.0
Millar (2008/8). Glossary of terms relating to capture–recapture methods
Starfield (2008/7). The concept of prevention: a good idea gone astray?
Bambra (2008/1). The welfare state: a glossary for public health

2007

Bambra (2007/12). The psychosocial and health effects of workplace reorganisation. 2. A systematic review of task restructuring interventions
De Mario (2007/10). Income inequality measures
Laurent (2007/08). Effect of socioeconomic status on the relationship between atmospheric pollution and mortality
Rutherford (2007/08). Violence: a glossary, and Violence: a priority for public health? (part 2)
Bambra (2007/07). A politics of health glossary
Whitehead (2007/06). A typology of actions to tackle social inequalities in health
Deary (2007/05). Cognitive epidemiology
Moore (2007/04). Public health preparedness: a systems-level approach
Decullier (2007/03). Oral presentation bias: a retrospective cohort study
Burger (2007/03). A glossary on psychiatric epidemiology
Rezaeian (2007/02).Geographical epidemiology, spatial analysis and geographical information systems: a multidisciplinary glossary
Tite (2007/01). Why do peer reviewers decline to review? A survey

2006

A glossary for the social epidemiology of work organisation

Benach (2006/11). Part 1, Terms from social psychology
Muntaner (2006/12). Part 2 Terms from the sociology of work and organisations
Hadden (2007/01). Part 3, Terms from the sociology of labour markets

Bray (2006/10). Advanced paternal age: How old is too old?
Baum (2006/10). Participatory action research
Whittaker (2006/10). Pragmatic randomised controlled trials in parenting research: the issue of intention to treat
Alvarez-Dardet (2006/08). Democratisation and health after the fall of the Wall
Susser (2006/08). A personal history: social medicine in a South African setting, 1952–5. Part 2: Social medicine as a calling: ups, downs, and politics in Alexandra Township
Labonte (2006/08). Glossary of the World Trade Organisation and public health: part 1
Labonte (2006/09). Glossary on the World Trade Organisation and public health: part 2
Carpiano (2006/07). A guide and glossary on postpositivist theory building for population health
Popay (2006/07). Whose theory is it anyway?
Dunn (2006/07). Speaking theoretically about population health
Vagero (2006/07). Where does new theory come from?
Carpiano (2006/07). Theory building on the high seas of population health: Love Boat, Mutiny on the Bounty, or Poseidon Adventure
Hernan (2006/07). Estimating causal effects from epidemiological data
Loue (2006/06). Community health advocacy
Eriksson (2006/05). Antonovsky’s sense of coherence scale and the relation with health: a systematic review
Waters (2006/04). Evaluating the effectiveness of public health interventions: the role and activities of the Cochrane Collaboration
Barreto (2006/03). Infectious diseases epidemiology
King (2006/03). What same sex civil partnerships may mean for health
Galobardes (2006/01). Indicators of socioeconomic position (part 1), and (part 2)

2005

Agyemang (2005/12). Labelling African origin populations in the health arena in the 21st
Nguyen (2005/11). Terms in reproductive and perinatal epidemiology: I. Reproductive terms
Nguyen (2005/12). Terms in reproductive and perinatal epidemiology: 2. Perinatal terms
Krantz (2005/10). Violence against women
Merlo. A brief conceptual tutorial on multilevel analysis in social epidemiology:

De Silva (2005/08). Social capital and mental illness: a systematic review
McKee (2005/08). International cooperation and health. Part I: issues and concepts and Part 2: making a difference
Baum (2005/07). Wealth and health: the need for more strategic public health research
Neutra (2005/07). Glossary of terms used when discussing exposure to electric and magnetic fields
Lindstrom (2005/06). Salutogenesis
Merlo (2005/06). A brief conceptual tutorial of multilevel analysis in social epidemiology
Krieger (2005/04). Embodiment: a conceptual glossary for epidemiology
Khawaja (2005/04). Poverty
Pless (2005/03). Injury prevention: a glossary of terms
Spencer (2005/02). Social paediatrics
Barnett (2005/02). Excess in cardiovascular events on Mondays: a meta-analysis and prospective study
McLaren (2005/01). Ecological perspectives in health research

2004

Hawe (2004/12). A glossary of terms for navigating the field of social network analysis
Halliday (2004/11). Genetics and public health—evolution, or revolution?
Regidor (2004/10). Measures of health inequalities: part 1 and part 2
Mackenbach (2004/09). Streets of Paris, sunflower seeds, and Nobel prizes. Reflections on the quantitative paradigm of public health
Kawachi (2004/09). Five American authors on wealth, poverty, and inequality
Almeida-Filho (2004/09). Saramago’s All the Names and the epidemiological dream
Berridge (2004/09). History of addictions
Delgado-Rodriguez (2004/08). Bias
Rychetnik (2004/07). A glossary for evidence based public health
Bhopal (2004/06). Glossary of terms relating to ethnicity and race: for reflection and debate
Chapman (2004/05). Advocacy for public health: a primer
Hernan (2004/04). A definition of causal effect for epidemiological research
Jefferson (2004/04). Peer review
Howden-Chapman (2004/03). Housing standards: a glossary of housing and health
Barker (2004/02). Developmental origins of adult health and disease
Rodriguez-Artalejo (2004/02). Cardiovascular epidemiology and prevention
Tunstall (2004/01). Places and health

2003

Illich (2003/12). Medical nemesis
Kuh (2003/10). Life course epidemiology
Mindell (2003/09). A glossary for health impact assessment
Malats (2003/06). Basic molecular genetics for epidemiologists
Malats (2003/07). Basic glossary on genetic epidemiology
Malats (2003/08). Advanced glossary on genetic epidemiology
Baum (2003/05). Social capital
Berridge (2003/03). Public health history
Garcia (2003/01). A glossary for research in occupational health

how to estimate the risk and relative risk from logistic regression

2017-08-09T12:50:00.003-04:00

How to estimate the risk and relative risk from logistic regression of a case-control study (OR to RR)
One advantage of odds ratios is that we can estimate it in a case-control study, when we usually oversample the cases. However, we cannot directly calculate the probability using logistic regression in a case-control study, since the beta(0) of the case-control study cannot represent the target population, which beta(0) of the target population is equal to beta(0) of the case-control study - log(sampling probability for cases/sampling probability for controls).

King (2004). Inference under full population information in "Case-Control Studies, Inference in" of Encyclopedia of Biopharmaceutical Statistics
Zhang (1998). What's the relative risk? A method of correcting the odds ratio in cohort studies of common outcomes
Greenland (1981). Multivariate estimation of exposure-specific incidence from case-control studies
Austin (2010). Overestimation of risk ratios by odds ratios in trials and cohort studies: alternatives to logistic regression
Austin (2011). A Tutorial on Methods to Estimating Clinically and Policy-Meaningful Measures of Treatment Effects in Prospective Observational Studies: A Review
Spiegelman (2005). Easy SAS calculations for risk or prevalence ratios and differences
Bartlett (2015). Estimating risk ratios from observational data in Stata
Chui (2013). A Regression-Based Method for Estimating Risks and Relative Risks in Case-Base Studies
Knol (2012). Overestimation of risk ratios by odds ratios in trials and cohort studies: alternatives to logistic regression

The Celiac Surge

2017-07-10T14:14:00.003-04:00

Article: The Celiac Surge
by Catherine Offord (TheScientist)
A rapid increase in the global incidence of the condition has researchers scrambling to understand the causes of the trend, and cope with the consequences. (Full Text)
Inforgraphic: Immune Irritation in the Gut

R! related

2017-04-05T11:10:00.046-04:00

R! related

Blog: R Documentation and Learning Resources

IDE and GUI

RStudio is an IDE for R. RStudio combines an intuitive user interface with powerful coding tools to help you get the most out of R. RStudio Webinars
Rcmdr is a GUI of R.
esquisse is a great addin for ggplot2
Deducer is a good but relative old GUI for exploring data like JMP with ggplot2 behind (Plot Builder). JGR is a Java GUI for R. (ggplot2 – much easier with JGR and Deducer). To use Deducer, you need - install.packages(c("JGR", "Deducer", "DeducerExtras")) -, submit: - library(JGR) -, - JGR() -; then, in the JGR console, to load Deducer, go to 'Packages & Data' > 'Package Manager' and select Deducer and DeducerExtras. more: all-in-one installing JGR and Deducer. Notes: 1) to set JAVA location using --- options(java.home="xxx/Java/") ---. 2) rJAVA ver 9.6 needs running under the 32 bit R! on my computers.
GrapheR (pdf) is another GUI for draw customized graphs without knowing any R commands.
Tessera - Open source environment for deep analysis of large complex data (Divide and recombine）
The application Bio7 is an integrated development environment for ecological modelling and contains powerful tools for model creation, scientific image analysis (ImageJ) and statistical analysis.

Communication between SAS and R

haven allows you to load foreign data formats (SAS, SPSS and Stata).

SAS calls functions in R language (pdf): RUN ExportDatasetToR(“xxx.mydata”); PROC IML; SUBMIT /R; ... R codes ... ENDSUBMIT; QUIT; RUN ImportDataSetFromR("WORK.xxx", "xxx");

Bewerunge (2011). Call R function from SAS (pdf)

Graphical parameters

Plot symbols

plot symbols (pch), size

Color

Pick a color site built in R with Shiny tags %$%

R Color Cheat Sheet, Color, palette (ggplot2: ggplot2 color, Cookbook for R), chart of R! colors.

RColorBrewer provides palettes for drawing nice maps shaded according to a variable.

Wesanderson provides palettes derived from the Tumblr blog Wes Anderson Palettes.

Simple Statistics: Colors in R

Glynn (2007). Using color in R

R! How can I include Greek letters in my plot labels?

Revolutions: How to make a heat map in R, Superheat: supercharged heatmaps for R

Chart Chooser — improves Excel and PowerPoint charts. there is R! version of Chart Chooser (not many charts on the site, but the idea is great)

Packages (rdrr.io, CRAN, Rdocumentation, Inside-R, Quick-R, Bioconductor)

CRAN Task View organizes the packages into different groups such as Graphics, Survival Analysis, etc.
margins: An introduction to 'margins'
Transition from Excel to R!:

DT: An R interface to the DataTables library
excelR: An R interface to jExcel library to create web-based interactive tables and spreadsheets compatible with Excel or any other spreadsheet software
DTedit: Editable DataTables for shiny apps
rhandsontable is a htmlwidget based on the handsontable.js library.
formattable is designed for applying formatting on vectors and data frames
rpivotTable is a R wrapper for the great library pivottable

Zelig is a general purpose statistics program fro estimating, interpreting, and presenting results from any statistical method. It turns the power of R with free ranging syntax, diverse examples, and documentation written for different audiences — into the same three commands and consistent documentation for every method
Report and documentation

knitr package was designed to generate dynamic report with R. Chunk options.
rmarkdown is an authoring format that enables easy creation of dynamic documents, presentations, and reports from R.
bookdown is to facilitate writing books and long-form articles/reports with R Markdown. Here are several very good free books published on the bookdown website

Programming

Devtools makes it so easy to build a package that it becomes your default way to organise code, data and documentation.

Data I/O and manipulation

data.table package provides an enhanced version of data.frame. Here is the manual and help. It's a star package on r-pkg.org.
rio is a Swiss-army knife for data I/O.
readr makes it easy to read many types of tabular data. -read_fwf()- is a powerful function for reading fixed width file, for example, read in the NCHS mortality file.
Janitor has simple functions for examining and cleaning dirty data
sqldf is an R package for running SQL statements on R data frames. sqldf supports SQLite database as default (SQLite syntax).

haven allows you to load foreign data formats (SAS, SPSS and Stata).

xlsx gives programatic control of Excel files using R.

read.table() reads a file in table format and creates a data frame from it.

arrow supports for analyzing large, multi-file datasets, working with individual Parquet and Feather files.

MMWRweek: Convert dates to MMWR Day, Week, and Year

dlookr: Diagnose, explore and transform data

foreign read data stored by other system.

dplyr is a new package which provides a set of tools for efficiently manipulating datasets in R. Introduction to dplyr. Notes: "Both data.table and dplyr were able to reduce the problem to less than a few seconds. If you’re looking for pure speed data.table is the clear winner. However, it is my understanding that data.table‘s syntax can be frustrating, so if you’re already used to the ‘Hadley ecosystem’ of packages, dplyr is a formitable alternative, even if it is still in the early stages."

tidyr with spread() and gather(), a reframing of reshape2, is a new package that makes it easy to “tidy” your data.
magrittr is a forward pipe operator. magrittr: Simplifying R code with pipes. History of Magrittr.
Aggregation and restructure: reshape2 package with melt() or cast() function.Transpose using t() function, aggregating data using aggregate() function.
lubridate is good for manipulating time and date.
Stringr makes it easier to work with strings (article).
Quick-R examples of outputing R data.
RStudio addins: datapasta, ggthemeassist
pdftools: Text Extraction, Rendering and Converting of PDF Documents (example)

Survey related packages

survey (Thomas Lumley) provides facilities in R for analyzing data from complex surveys. Complex sampling and R (pdf). Analyze US Government Survey Data with R. You can use "survey" to create replicate weights, and use withReplicates computes variances by replicate weighting.
rpms is for Recursive Partitioning for Modeling Survey data
lavaan.survey is for complex survey structural equation modeling (SEM). An R Package for Complex SurveyAnalysis of Structural Equation Models (pdf)
sampling includes many different algorithms for drawing survey samples and calibrating the design weights.
convey is a package for estimating indicators of income concentration and poverty (including gini coefficient, atkinson index, at-risk-of-poverty threshold etc.) wrapped around the Lumley's survey package by Anthony Damico. He has worked "analyze survey data for free with R language" for years (asdfree).

GIS

Pebesma: Spatial Data Science with application in R
Methods Bites: Using Geospatial Data in R
Pebesma: Simple Features for R
R-spatial.org: Drawing beautiful maps programmatically with R, sf and ggplot2, Part 1, Part 2, Part 3
geofacet: Introduction to geofacet
ggmap is the easy visualization of spatial data and models on top of Google Maps, OpenStreetMaps, Stamen Maps, or CloudMade Maps using ggplot2. German Gas Prices illustrated, Cheatsheet

Robin. Introductory tutorial on graphical display of geographical information in R
Other packages: maps (draw geographical maps), maptools (reading and hadling spatial objects), mapdata (extra map databses), scales (scale functions for visualization), mapproj (map projections), RgoogleMaps (overlays on static maps)

tidycensus is an R package that allows users to interface with the US Census Bureau’s decennial Census and five-year American Community APIs

Data Visualization and Graphics

The R graph Gallery
Auguie: Laying out multiple plots on a page
patchwork: to arrange your graphs easily

ggplot2 (document) (You can build the book here: ggplot2: Elegant graphics for data analysis) is a plotting system for R, based on the grammar of graphics, which tries to take the good parts of base and lattice graphics and none of the bad parts. Cookbook for R is a great book. Tutorial websites: Tutorial I, Tutorial II. A good book website of Guide to Create Beautiful Graphics in R (Cheatsheet: Be Awesome in ggplot2). Beautiful plotting in R: A ggplot2 cheatsheet (pdf). Tutorial video (Part I and Part II) by Roger Peng

ggplot2 extensions gallery
ggThemeAssist is a RStudio-Addin and provides a GUI for editing ggplot2 themes
esquisse is a addin, which allows you to interactively explore your data by visualizing it with the ggplot2 package.
cowplot and ggstatsplot are add-on packages of ggplot2
ggpubr provides some functions for creating figures based on publication requirement
You may use ggplot_gtable() to plot data generated by ggplot_build() to find the plot objects (grob) (grobs <- font="" ggplot_build="" ggplot_gtable="" yourgraph="">), then use the grid.draw() of grid package to draw them individually. Hadley Wickham has blogs about how to Mixing ggplot2 graphs with other graphical output, Editing raw Grid objects from a ggplot.
ggedit is used to facilitate ggplot formatting

A conversation with Hadley Wickham 2014 - Eduardo
ggthemes is a package including some extra geoms, scales, and themes styles (the Economist, Excel, Stata, etc.) for ggplot2.
GGally is designed to be a helper to ggplot2. It contains templates for different plots to be combined into a plot matrix, a parallel coordinate plot function, as well as a function for making a network plot.
gtable is a package of tools to make it easier to work with ``tables'' of grobs, which is internally used by ggplot2. I used the gtable::cbind(..., size = "max", z = NULL) or gtable::rbind(..., size = "max", z = NULL) to match the size of plots.
ggprepel is a ggplot2 extension to avoid overlapping text labels.
Walker (2014). International population pyramids with ggplot2

ggforce accelerates 'ggplot2'
ggraph is an extension of ggplot2 tailored at plotting graph-like data structures (graphs, networks, trees, hierarchies...)
WVPlots Provides examples of ggplot plots that can be generated from a standard calling interface
classifierplots generates a visualization of binary classifier performance as a grid of diagonstic plots with just one function call
ggalt: amke a dumbbell plot in ggplot2

ggcharts aims to get you to your desired plot faster.
Plotly: ggplotly of plotly package is a powerful tool to convert ggplot2 plots and create interactive, online ggplot2 charts with D3.js. Carson Sievert used this converter recreated Hadley Wickam’s entire ggplot2 documentation (here). Click-drag to zoom, shift-click to pan, double-click to autoscale. It's really amazing. Plotly Tutorial: Plotly and R

grid and gridExtra are useful combined with ggplot2, for example:

library(grid) # for unit() in ggplot2
grid.arrange(p1, p2, p3, p4,ncol=2)

ggvis new package for data visualization. Like ggplot2, it is built on concepts from the grammar of graphics, but it also adds interactivity, a new data pipeline, and it renders in a web browser.
vcd for visualizing categorical data. Working with categorical data with R and the vcd and vcdExtra packages and Visualizing Categorical Data with SAS and R by Michael Friendly.
Distribution visualization: vioplot for violin plot (examples of boxplot and violin plot). Hintze (1998). Violin Plots: A Box Plot-Density Trace Synergism. Violin plot using ggplot2. Boxplots and beyond: Part I, II: asymmetry, III: violin plots, IV: beanplots.
venneuler is my favorite package for make a area proportional Venn and Euler Diagram simple and fast, I use it create diagrams then use Inkscape to modify them. A more complicated package is VennDiagram. Wilkinson (2012) introduced his venneuler package in the artilce "Exact and Approximate Area-proportional Circular Venn and EulerDiagrams". Chen (2011) compared some R! package and programs in the article "VennDiagram: a package for the generation of highly-customizable Venn and Euler diagrams in R" (Create a Venn diagram using MS Excel)

venn = venneuler(c(a1c=.15, fpg=.11, ogtt=.13, 'a1c&fpg'=.1, 'fpg&ogtt'=.13, 'a1c&ogtt'=.4, 'a1c&ogtt&unknow'=.1))
plot(venn)

Vennerable package provides routines to compute and plot Venn diagrams, including the classic two- and three-circle diagrams but also a variety of others with different properties such as rectangular or square Venn diagram and for up to seven sets. It's not available at CRAN. You can install this package using: install.packages("Vennerable", repos = "http://R-Forge.R-project.org")
googleVis - Google Motion Charts with R!
colourlovers provides access to the COLOURlovers API, which offers color inspiration and color palettes
Lattice is a powerful and elegant high-level data visualization system, with an emphasis on multivariate data.
scatterplot3d provides routines for the visualization of multivariate data in a three dimensional space.
iPlots is a package for the R! which provides high interaction statistical graphics, written in Java.
rcharts is an R package to create, customize and publish interactive javascript visualizations from R using a familiar lattice style plotting interface.
wordcloud2 and wordcloud create wordcloud for data visualization. Using R to create a Word Cloud from a PDF Document. 可能是目前最好的词云解决方案wordcloud2
David Smith (2017).Packages to simplify mapping in R
magick is a package of wrapping up of ImageMagick which is an image processing library
networkD3: D3 JavaScript Network Graphs from R, good at Sankey Diagram

Missing Imputation

Amelia: A Program for Missing Data (missmap())
Berk (2021): Handling missing data using XGBoost, XGBoost
Josse: Handling missing values with R
MissingDataGUI: a GUI for missing data exploration
MICE: multiple imputation using fully conditional specification.

Statistics/Machine Learning

MLR – Machine Learning in R and MLR3 a scussor of MLR. Here is the mlr3 book
tidymodes: a combination of multiple packages: parsnip, recipes, etc.

tidyverse is a collection of popular packages for data munging from Hadley Wickham including ggplot2, dplyr, tidyr, readr, purrr, tibble, etc.

bayestestR provides a comprehensive and consistent set of functions to analyze and describe posterior distributions generated by a variety of models objects, including popular modeling packages such as rstanarm, brms or BayesFactor.
rms: is a package goes along with the book Regression Modeling Strategies
caret is a set of functions that attempt to streamline the process for creating predictive models. Here are caret webinar and slides
splines is now a part of R! standard/add-on packages/bundle. splines::ns() generate a matrix for natural cubic splines. Bendix tuned the ns() as Ns() in his Epi package, which used the smallest and the largest of the supplied knots as boundary knots. You can download the latest version on his website here.
lavaan: latent variable analysis
msm::deltamethod: delta method, FAQ UCLA
nls can be used to determine the nonlinear (weighted) least-squares estimates of the parameters of a nonlinear model, which is similar to the 'nl' of stata and 'proc nlin' of SAS
rpart is for the recursive partitioning for classification, regression and survival trees.

Therneau (2015).An introduction to recursive partitioning using the RPART routines
Classification Trees using the rpart function (2010)
Let's have a "party" and tear this place "rpart"! (2013)

Hmisc: Harrell Miscellaneous
convey estimates measures of poverty, inequality, and wellbeing using the complex survey data
deltavar() is a function in the emdbook packages of the book, which calculates delta-method-based variances for functions with any number of parameters

deltavar(log(A/B),meanval=c(A=0.8,B=12),Sigma=matrix(c(0.1,0,0,8.0),nrow=2)) ---> 0.2118056 = ((1/A)^2*Var(A)+(1/B)^2*Var(B))^0.5
deltavar(log(A/B),meanval=c(A=0.8,B=12),Sigma=c(0.1,8.0)) -> 0.2118056

MASS is a classic package for Venables and Ripley's Modern Applied Statistics with S
Bolstad is A set of R functions and data sets for the book Introduction to Bayesian Statistics, Bolstad, W.M. (2016). The "bayes.lin.reg" function may be used for combining the two estimates like meta-analysis.
Psych: Using R for psychological research

Electronic Health/Medical Record (EHR) related

coder: Deterministic Categorization of Items Based on External Code Data by rOpenSci (blog)

R functions and keyboard shortcuts

2017-03-24T14:43:00.000-04:00

R functions/commands and keyboard shortcuts

R! is powerful and has rich packages and functions. It's impossible to build a list of functions/shortcuts to fit the purposes of all. Below are some functions/shortcuts related to my projects.

Cheatsheets, The R Guide, R Reference Card
Help functions: help()/?, apropos(), find(): apropos() finds all objects. find() the locations of found objects, methods(), example(), demo(), vignette(), args()
Housekeeping functions: getwd(), setwd(), rm(list=ls()) removes all objects in the R environment, source("myRscript.r") runs the R codes in "myRscript.r" file, fix() modifies the original object, and edit() is used edit an object and returns to a new object, download.file() downloads a file from the Internet, attach()/detach() objects, search() shows the current search paths and sequence, install.packages(), update.packages(), remove.packages(), getOption("defaultPackages") which can be changed by setting the option in startup code (e.g. in ~/.Rprofile), .libPaths()

Numeric/character functions: length(), seq(), rep(), cut(), pretty(), cat(), substr(), grep(), sub(), strsplit(), paste(), toupper(), tolower()
Data functions: read.table(), head(), tail(), str(), class(), length(), dim(), nrow(), ncol(), names(), levels(), length(), c(), cbind(), rbind(), append(), rep(), rev(), sort(), unique()
Type functions: "is." for checking or "as." for conversion + numeric(), character(), vector(), matrix(), data.frame(), factor(), logical(), integer(). For example: is.numeric(), as.numeric()
Mathematical functions: abs(), sqrt(), log(), log(x, base=n), log10(), exp(), prod(), factorial(), choose(), ceiling(), floor(), solve(), trunc(), round(), signif(), cos(), sin(), tan(), acos()
Statistical functions: mean(), median(), sd(), var(), mad(), quantile(), range(), sum(), diff(), min(), max(), scale(), fivenum(), cumsum(), cumprod(), cummax(), cumin(), cor(), colSums(), rowSums(), colMeans(), rowMeans()
Probability functions: the form is [d][p][q][r]distribution(). d, p, q, r are for (d)ensity, cumulated (p)robability/distribution function, (q)uantile function, and (r)andom generation, respectively. the Distribution types can be: (norm)al, (beta), (binom)ial, (chisq)uared, (exp)onential, (logis)tic, (multinom)ial, (n)egative (binom)ial, (pois)son, (f), (gamma), (t), (unif)orm, etc. for example: dnorm(), pnorm(), qnorm(), rnorm()
Statistical modeling functions

Model functions: lm(), glm(), nls(), nls2(), lme() / nlme()
Symbol formulas (y ~ A + B + C ): ":" is for interaction term, "*" is for complete interaction, "^" is for crossing to a specified degree "." is a placeholder for all other variables except the dependent variable, "-" removes a variable from the equation, "-1" suppresses the intercept, "I()" has elements within the parentheses interpreted arithmetically

Post-estimation functions: coef(), confint(), resid(), fitted(), summary(), predict(), deviance(), print(),plot(), formula(), anova(obj1, obj2), AIC(), vcov()
Contrast functions: contr.helmert(), contr.poly(), contr.sum(), contr.treatment(), contr.SAS()

RStudio is an integrated development environment (IDE) for R. RStudio combines an intuitive user interface with powerful coding tools to help you get the most out of R. Shortcuts (you can modify them: Tools -> Modify Keyboard Shortcuts...)

Alt + Shift + K: Show a Quick Reference
Alt + -: Insert assignment operator "<- font="">
Ctrl + Shift + M: Insert pipe operator "%>%" (I changed it as Ctrl + Shift + P)
Ctrl + Alt + I: Insert chunk (R Notebook/Markdown)
Ctrl + 1: Move cursor to source Editor window
Ctrl + 2: Move cursor to Command window
Ctrl + 3: Move cursor to Help window
Ctrl + 4: Move cursor to History window
Ctrl + 5: Move cursor to File window
Ctrl + 6: Move cursor to Plots window

choice of analytical language

2017-03-13T16:22:00.000-04:00

Choice of analytical language
I have used mainly three statistical languages, Stata, R, and SAS, for many years for different purposes. The weights of usage of those three languages are shift from SAS-Stata-R to SAS-R-Stata, then, to Stata-R-SAS. Sometimes I am asked to recommend a better analytic language, which is always a hard and complicated question to me. I came across an blog written by Curtis Miller, which is very thoughtful and helpful to make this kind of choice. Here is his blog: "On Programming Languages; Why My Dad Went From Programming to Driving a Bus". Hopefully his story can help you to make your own decision.

Stata News: in the spotlight

2017-03-08T10:20:00.008-05:00

Stata News: in the spotlight etc.

2021

2020

2019

Customized forest plots for displaying meta-analysis results
Importing data from SPSS and SAS
Fun with frames
Lasso
Interpreting models for log-transformed outcomes (unbiased prediction: E(Y|X) = e^XBe^σ²/2 )
User's corner: ftools and gtools

2018

2017

In the spotlight: Nonlinear multilevel mixed-effects models
Cheatsheet: User's Corner: Quick references for your favorite commands
What's new in Stata 15 (released on 2017-06-06, 15.1 released on 2017-12-20)
Visualizing continuous-by-continuous interactions with margins and twoway contour

2016

2015

Easy-to-interpret, flexible survival-time treatment effects, and Postestimation Selector
Treatment effects, and irt
Bayesian “random-effects” models, and What's New in Stata 14
Finding and using results, constants, functions ... anything (Data > Other utilities > Hand calculator), and forecast for dynamic panel data and counterfactuals

2014

2013

2012

2011

2010

Syndemics: health in context

2017-03-03T14:20:00.001-05:00

Syndemics: health in context
A syndemic, coined by Merrill Singer in mid-1990s, is a conceptual framework for understanding diseases or health conditions that arise in populations and that are exacerbated by the social, economic, environmental, and political milieu in which a population is immersed. The today's issue of Lancet published a series related the syndemic... full text ...

Information about the Global Burden of Diseases, Injuries, and Risk Factors Study

2017-01-24T16:33:00.001-05:00

Information about the Global Burden of Diseases, Injuries, and Risk Factors Study

WHO: the Global Burden of Disease (GBD) project
Lancet Global Burden of Disease
UW website directed Christopher Murray
Wikipedia: History of GDB
List of causes and ICD9 and ICD10 of these causes in eTable 2 of supplement
Search title with "Global Burden of Disease Study" on the PubMed