Fellgernon Bit - rstats

Cleaning up my R packages and config files

Tue, 03 Nov 2020 00:00:00 +0000

Today was an unusual day at work given the US Elections. This meant that I had fewer meetings than what I’ve had lately. Earlier in the day I noticed an email announcing that the Bioconductor 3.13 docker image had been released for the next 6 month development cycle, which was a reminder of the recent Bioconductor 3.12 release. This prompted me to start updating my R packages.

In the past, I’ve updated all my currently installed R packages using the framework I described in a 2017 blog post. I remember seeing a tweet by Hadley Wickham not so long ago ¹ that for him, a new R version was an opportunity to start with a clean slate. I like having everything I need ready to use, but well, my list of installed R packages was getting pretty long. Given that 4 year windows of time are in our minds, it felt like a good opportunity to clean my house. Or well, my R packages.

Thus, I started writing down which are the packages I want to have installed. At this point for me, that includes several R/Bioconductor packages I’ve made and their dependencies in case I need to work on them to resolve bugs or add new features. My R packages already use many of my favorite R packages, so I took advantage of this in order to avoid having to list every single R package I like using. In order to achieve this, I used the dependencies = TRUE argument that you can use with remotes and BiocManager.

## Install from scratch
if (!requireNamespace("remotes", quietly = TRUE))
    install.packages("remotes")
remotes::install_cran("BiocManager")
BiocManager::version()

## Rprofile packages
remotes::install_github(c(
    "jalvesaq/colorout"
))
remotes::install_cran(c(
    "devtools",
    "usethis"
))

## Main packages
BiocManager::install(c(
    "biocthis",
    "brainflowprobes",
    "derfinder",
    "derfinderPlot",
    "GenomicState",
    "megadepth",
    "recount",
    "recountWorkflow",
    "recount3",
    "regutools",
    "regionReport",
    "spatialLIBD"
), dependencies = TRUE, update = FALSE)

Once I had my main packages, I started adding some from LIBD, some from CRAN, and other ones from Bioconductor. You can see the full list at my team’s website under Config files: R setup; R packages.

I was curious about how these changes affected my list of installed R packages and used my older 2017 blog post code to check this. That resulted in this list which shows 423 installed R packages and 589 that I used to have installed. I suspect that several of them will come back. For example, I needed to install blogdown to work on this blog post. Some of the 423 packages are new, like rsthemes which we recently learned about at the LIBD rstats club.

Today @lcolladotor went over new @rstudio & @Bioconductor developments that was started by tweets by @apreshill & @grrrck + new #rstats 📦s #recount3 #megadepth involving @chrisnwilks @dyzhang32 et al

📹https://t.co/E0DZ9Ej5qV
📓 https://t.co/FnDvcUmyUO
👀https://t.co/mEbQirhkF6 pic.twitter.com/BWByp9W7Wh
— LIBD rstats club (@LIBDrstats) October 3, 2020

Config files

Since I was doing all this work on both my macOS and Windows laptops for my R setup, I also went ahead and cleaned up a bit my configuration files. I have several of them with settings that I recommend others to use. That’s why I wrote a little “chapter” about them on my team’s website. The list includes:

Software I use (including R and RStudio)
R packages
R configuration files such as ~/.Rprofile and ~/.Renviron
Git configuration files ~/.gitconfig and ~/.gitignore_global
JHPCE (linux) configuration files such as ~/.bashrc, ~/.inputrc, ~/.bash_profile and ~/.sge_request

Wrapping up

I’m hoping that all this information will be useful to both current and new team members, but it could be useful also to you. Though you might need to adapt some things. Earlier on in my career I learned from how others use configuration files to speed up their work or make their work experience more enjoyable. I’m still learning, but now I have a decent bag of tricks to share too.

Have fun!

Image source

Acknowledgments

This blog post was made possible thanks to:

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2020. URL: https://github.com/cboettig/knitcitations.

[2] G. CsÃ¡rdi, R. core, H. Wickham, W. Chang, et al. sessioninfo: R Session Information. R package version 1.1.1. 2018. URL: https://CRAN.R-project.org/package=sessioninfo.

[3] A. Oles, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.18.0. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## - Session info -------------------------------------------------------------------------------------------------------
##  setting  value                       
##  version  R version 4.0.3 (2020-10-10)
##  os       Windows 10 x64              
##  system   x86_64, mingw32             
##  ui       RTerm                       
##  language (EN)                        
##  collate  English_United States.1252  
##  ctype    English_United States.1252  
##  tz       America/New_York            
##  date     2020-11-03                  
## 
## - Packages -----------------------------------------------------------------------------------------------------------
##  package       * version date       lib source                                 
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 4.0.3)                         
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 4.0.3)                         
##  BiocStyle     * 2.18.0  2020-10-27 [1] Bioconductor                           
##  blogdown        0.21    2020-10-11 [1] CRAN (R 4.0.3)                         
##  bookdown        0.21    2020-10-13 [1] CRAN (R 4.0.3)                         
##  cli             2.1.0   2020-10-12 [1] CRAN (R 4.0.3)                         
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 4.0.3)                         
##  digest          0.6.27  2020-10-24 [1] CRAN (R 4.0.3)                         
##  evaluate        0.14    2019-05-28 [1] CRAN (R 4.0.3)                         
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 4.0.3)                         
##  generics        0.1.0   2020-10-31 [1] CRAN (R 4.0.3)                         
##  glue            1.4.2   2020-08-27 [1] CRAN (R 4.0.3)                         
##  htmltools       0.5.0   2020-06-16 [1] CRAN (R 4.0.3)                         
##  httr            1.4.2   2020-07-20 [1] CRAN (R 4.0.3)                         
##  jsonlite        1.7.1   2020-09-07 [1] CRAN (R 4.0.3)                         
##  knitcitations * 1.0.10  2020-11-03 [1] Github (cboettig/knitcitations@ea5d202)
##  knitr           1.30    2020-09-22 [1] CRAN (R 4.0.3)                         
##  lubridate       1.7.9   2020-06-08 [1] CRAN (R 4.0.3)                         
##  magrittr        1.5     2014-11-22 [1] CRAN (R 4.0.3)                         
##  plyr            1.8.6   2020-03-03 [1] CRAN (R 4.0.3)                         
##  R6              2.5.0   2020-10-28 [1] CRAN (R 4.0.3)                         
##  Rcpp            1.0.5   2020-07-06 [1] CRAN (R 4.0.3)                         
##  RefManageR      1.3.0   2020-11-03 [1] Github (ropensci/RefManageR@ab8fe60)   
##  rlang           0.4.8   2020-10-08 [1] CRAN (R 4.0.3)                         
##  rmarkdown       2.5     2020-10-21 [1] CRAN (R 4.0.3)                         
##  sessioninfo   * 1.1.1   2018-11-05 [1] CRAN (R 4.0.3)                         
##  stringi         1.5.3   2020-09-09 [1] CRAN (R 4.0.3)                         
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 4.0.3)                         
##  withr           2.3.0   2020-09-22 [1] CRAN (R 4.0.3)                         
##  xfun            0.19    2020-10-30 [1] CRAN (R 4.0.3)                         
##  xml2            1.3.2   2020-04-23 [1] CRAN (R 4.0.3)                         
##  yaml            2.2.1   2020-02-01 [1] CRAN (R 4.0.3)                         
## 
## [1] D:/R/R-4.0.3/library

I couldn’t find the tweet right now.↩︎

The start of a new phase for me

Mon, 21 Sep 2020 00:00:00 +0000

Today I have accepted a new role at the Lieber Institute for Brain Development (LIBD) as an Investigator. Since LIBD is affiliated with Johns Hopkins University (JHU), LIBD will support me on becoming a JHU faculty member ¹. This change means that I’ll now be leading a team at LIBD, which is why I’m actively working on a new website ² called R/Bioconductor-powered Team Data Science. On that website I’m implementing the ideas I recently proposed on another type of data science group which involve Data Science guidance sessions based on my three year experience as a teaching assistant for the MPH capstone at JHBSPH.

I also plan to make publicly available as much material as we can. This will include a series of bootcamps that I’m going to start running today for my team members and guests. It will also include our team journal/software club presentations as well as the LIBD rstats club presentations. The material that I’ll be using was made by many collaborators and friends at JHU, Harvard, WEHI, RStudio and the rstats community in general. I’m really thankful that so many people have shared their educational materials in ways that can be re-used and for making it easier for all of us to learn from them. Thank you!

Beyond the 20% education and data science guidance efforts, my team will be working with our LIBD peers, JHU collaborators, and external allies on interesting research projects such as the teamSpatial ³ that developed #spatialLIBD, #recount3, and many other upcoming projects in the realms of RNA-seq, R/Bioconductor, and genomics. We will be developing methods and software, analyzing data, and aim to build a stronger bridge between LIBD and JHU.

I’m really thankful to everyone who has been a part of my journey: mentors, colleagues, allies, sponsors, and many friends. I want to highlight my mentors Jeff Leek and Andrew Jaffe without whom I wouldn’t be where I am right now. They’ve always been my fiercest advocates and have looked out for me all these years. It’s now my turn to try have a positive influence in others, though they set a very high bar! I have a lot to learn and work to do, but I’m looking forward to it. At LIBD, JHU and elsewhere I have many people I can ask for support, which I’m grateful for.

Finally, after reading Jeff Leek’s How to be a modern scientist book in preparation for the first bootcamp today, I’ve updated my Twitter username to match my GitHub one. That is, I’ve gone from fellgernon to lcolladotor. See you around on Twitter and elsewhere!

P.S. If you find this type of team and the work that we do appealing, let me know and we’ll evaluate what options are available =)

Acknowledgments

This blog post was made possible thanks to:

R/Bioconductor-powered Team Data Science was made possible thanks to bookdown and a GitHub Actions workflow that depends on Bioconductor’s docker image, which in turn depends on the Rocker Project.

bookdown (Xie, 2016)

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] G. Csárdi, R. core, H. Wickham, W. Chang, et al. sessioninfo: R Session Information. R package version 1.1.1. 2018. URL: https://CRAN.R-project.org/package=sessioninfo.

[3] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.17.0. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[4] Y. Xie. bookdown: Authoring Books and Technical Documents with R Markdown. ISBN 978-1138700109. Boca Raton, Florida: Chapman and Hall/CRC, 2016. URL: https://github.com/rstudio/bookdown.

[5] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 4.0.2 (2020-06-22)
##  os       macOS Catalina 10.15.6      
##  system   x86_64, darwin17.0          
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-09-21                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 4.0.0)                    
##  bibtex          0.4.2.2 2020-01-02 [1] CRAN (R 4.0.0)                    
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 4.0.0)                    
##  BiocStyle     * 2.17.0  2020-05-14 [1] Bioconductor                      
##  blogdown        0.20    2020-06-23 [1] CRAN (R 4.0.2)                    
##  bookdown        0.20    2020-06-23 [1] CRAN (R 4.0.2)                    
##  cli             2.0.2   2020-02-28 [1] CRAN (R 4.0.0)                    
##  colorout      * 1.2-2   2020-05-18 [1] Github (jalvesaq/colorout@726d681)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 4.0.0)                    
##  digest          0.6.25  2020-02-23 [1] CRAN (R 4.0.0)                    
##  evaluate        0.14    2019-05-28 [1] CRAN (R 4.0.0)                    
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 4.0.0)                    
##  generics        0.0.2   2018-11-29 [1] CRAN (R 4.0.0)                    
##  glue            1.4.2   2020-08-27 [1] CRAN (R 4.0.2)                    
##  htmltools       0.5.0   2020-06-16 [1] CRAN (R 4.0.2)                    
##  httr            1.4.2   2020-07-20 [1] CRAN (R 4.0.2)                    
##  jsonlite        1.7.1   2020-09-07 [1] CRAN (R 4.0.2)                    
##  knitcitations * 1.0.10  2019-09-15 [1] CRAN (R 4.0.0)                    
##  knitr           1.29    2020-06-23 [1] CRAN (R 4.0.2)                    
##  lubridate       1.7.9   2020-06-08 [1] CRAN (R 4.0.2)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 4.0.0)                    
##  plyr            1.8.6   2020-03-03 [1] CRAN (R 4.0.0)                    
##  R6              2.4.1   2019-11-12 [1] CRAN (R 4.0.0)                    
##  Rcpp            1.0.5   2020-07-06 [1] CRAN (R 4.0.2)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 4.0.0)                    
##  rlang           0.4.7   2020-07-09 [1] CRAN (R 4.0.2)                    
##  rmarkdown       2.3     2020-06-18 [1] CRAN (R 4.0.2)                    
##  sessioninfo   * 1.1.1   2018-11-05 [1] CRAN (R 4.0.0)                    
##  stringi         1.5.3   2020-09-09 [1] CRAN (R 4.0.2)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 4.0.0)                    
##  withr           2.2.0   2020-04-20 [1] CRAN (R 4.0.0)                    
##  xfun            0.17    2020-09-09 [1] CRAN (R 4.0.2)                    
##  xml2            1.3.2   2020-04-23 [1] CRAN (R 4.0.0)                    
##  yaml            2.2.1   2020-02-01 [1] CRAN (R 4.0.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/4.0branch/Resources/library

The rules are Department and School specific at JHU, which is something that I have yet to explore in detail.↩︎
It’s technically a book made using bookdown.↩︎
Keri Martinowich, Kristen Maynard, Stephanie Hicks, and their team members.↩︎

Research Scientist: an academic career launch pad

Mon, 16 Mar 2020 00:00:00 +0000

After a long start to 2020 including the past four very busy weeks, I’m happy to announce that today March 16th 2020 I accepted a position as Research Scientist at the Lieber Institute for Brain Development in Baltimore, MD, USA.

via GIPHY

What will I do as a Research Scientist at LIBD?

At LIBD we currently have the following scientific ranks:

Research Technician
Research Assistant
Research Associate
Staff Scientist I, II and III
Research Scientist (+ Lead and Senior)
Investigator (+ Lead and Senior)

Research Scientists carry out research, do so scholarly, are tasked with being creative, are encouraged to seek funding, and can have supervisory and mentor roles.

In my agreement with LIBD, I will remain affiliated with the Data Science Team I led by Andrew E Jaffe while also having the opportunity to interact with and collaborate with fantastic biologists, data scientists, and researchers at LIBD, Johns Hopkins University, and beyond. Furthermore, I will officially help mentor LIBD scientists in data science and R tools. LIBD will support me while I build a small team around my research interests and prepare my first set of grant submissions. There will be a transition period and many things I need to learn: from requesting a budget to writing a job ad, as well as many details about grant requests. But ultimately, LIBD offered me a launch pad for an academic career.

via GIPHY

Details

One of my first tasks will be to refine the details of my role. For example, right now on the teaching side of things I’m imagining doing the following:

Lead a weekly LIBD rstats club meeting where I can go over topics common to several users.
Hold one-on-one data science guidance sessions, similar to the MPH capstone sessions I did during my teaching assistant years at JHBSPH.
Occasionally lead internal workshops, which could benefit from workshops I prepare for CDSB and elsewhere.
Get certified by RStudio and attend more R conferences & workshops.

From the scientific side, I will also learn more about the biological questions my LIBD colleagues are working on. These hypothesis-driven projects will provide me a new set of challenges and I will perform what me and my LCG-UNAM classmates were trained to do: to function as a bridge between multiple fields. I will also keep learning from my JHU colleagues and working with them as I refine my ideas. Currently, this is how I have framed my research interests:

My research aims to better understand the roots and signatures of disease (particularly psychiatric disorders) by zooming in across dimensions of gene activity: from studying gene expression at all feature levels (genes to exons to exon-exon junctions and un-annotated regions of expression), to using different gene expression measurement technologies (bulk RNA-seq, single cell/nuclei RNA-seq to spatial transcriptomics) that provide finer biological resolution and localization of gene expression. I’m interested in both hypothesis-driven projects as well as building general resources such as recount2 that enable us to contextualize our findings across all of the public human gene expression landscape. I use the R programming language for nearly all my work and like to organize my code in R packages that I share mostly through the Bioconductor project.

Marvin Meyer

As for building my team, I know that overall one of the challenges everyone faces is recruiting talent. I’ll give this complicated process a shot by tapping into my network which includes many immigrants. I will also expand my network online and through conferences. Furthermore, I am committed to my outreach projects such as CDSB but will synergize them even more: re-use training materials, potentially recruit team members, and help me expand my network. In the meantime, if you are interested in working with me or my colleagues, please let me know!

Final decision

The decision I had to make was challenging, but in the end, I decided to stay at LIBD because they offered me a path to become a principal investigator in academia which it’s something I want to explore and see where it leads me. I will also formally be someone’s boss and gain managerial experience, work much closer with biologists in hypothesis-driven projects, and overall gain more experience and skills. My brother’s job tasks changed last week, to which I told him: “welcome to the real world”. Well, I guess that’s how it’ll be for me too.

Acknowledgments

I want to thank everyone that helped me through this decision-making process, provided me with information, asked me questions, answered questions, expanded my universe, advocated for me, advised me, cheered for me, and supported me. Thank you! I look forward to working with you!

via GIPHY

This blog post was made possible thanks to:

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] G. Csárdi, R. core, H. Wickham, W. Chang, et al. sessioninfo: R Session Information. R package version 1.1.1. 2018. URL: https://CRAN.R-project.org/package=sessioninfo.

[3] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.14.4. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.3 (2020-02-29)
##  os       macOS Catalina 10.15.3      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-03-17                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.6.0)                    
##  bibtex          0.4.2.2 2020-01-02 [1] CRAN (R 3.6.0)                    
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 3.6.1)                    
##  BiocStyle     * 2.14.4  2020-01-09 [1] Bioconductor                      
##  blogdown        0.18    2020-03-04 [1] CRAN (R 3.6.0)                    
##  bookdown        0.18    2020-03-05 [1] CRAN (R 3.6.0)                    
##  cli             2.0.2   2020-02-28 [1] CRAN (R 3.6.0)                    
##  colorout      * 1.2-1   2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.6.0)                    
##  digest          0.6.25  2020-02-23 [1] CRAN (R 3.6.0)                    
##  evaluate        0.14    2019-05-28 [1] CRAN (R 3.6.0)                    
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  glue            1.3.2   2020-03-12 [1] CRAN (R 3.6.0)                    
##  htmltools       0.4.0   2019-10-04 [1] CRAN (R 3.6.0)                    
##  httr            1.4.1   2019-08-05 [1] CRAN (R 3.6.0)                    
##  jsonlite        1.6.1   2020-02-02 [1] CRAN (R 3.6.0)                    
##  knitcitations * 1.0.10  2019-09-15 [1] CRAN (R 3.6.0)                    
##  knitr           1.28    2020-02-06 [1] CRAN (R 3.6.0)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.6.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.6.0)                    
##  plyr            1.8.6   2020-03-03 [1] CRAN (R 3.6.2)                    
##  R6              2.4.1   2019-11-12 [1] CRAN (R 3.6.1)                    
##  Rcpp            1.0.3   2019-11-08 [1] CRAN (R 3.6.0)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.6.0)                    
##  rlang           0.4.5   2020-03-01 [1] CRAN (R 3.6.0)                    
##  rmarkdown       2.1     2020-01-20 [1] CRAN (R 3.6.0)                    
##  sessioninfo   * 1.1.1   2018-11-05 [1] CRAN (R 3.6.0)                    
##  stringi         1.4.6   2020-02-17 [1] CRAN (R 3.6.0)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.6.0)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.6.0)                    
##  xfun            0.12    2020-01-13 [1] CRAN (R 3.6.0)                    
##  xml2            1.2.5   2020-03-11 [1] CRAN (R 3.6.0)                    
##  yaml            2.2.1   2020-02-01 [1] CRAN (R 3.6.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

Diving together into the unknown world of spatial transcriptomics

Sat, 29 Feb 2020 00:00:00 +0000

Yesterday was an extremely exciting day for me and my colleagues. We finished a project we had been working on and shared it with the world. Meaning, it’s done and we can relax for a little bit while we wait for feedback from our peers.

But this was not any project, at least not for me. Why do you ask? In general terms, it involved an analysis that you could not search on Google and find the answer for. That is, it involved diving into the unknown!

via GIPHY

The unknown is scary and as the lyrics say:

I’ve had my adventure, I don’t need something new I’m afraid of what I’m risking if I follow you Into the unknown

All of us have been building our careers with other types of data and/or experiments, and taking on a new type of data knowing we had an early access advantage over others was quite the challenge. I don’t know about my co-authors, but maybe some of them shared thoughts like mine that were along the lines: can I do this? can I make it work? do my analysis choices make sense? what will experts think of doing once they have access to this data? All while racing against time, even if it was just an illusion in our minds.

But it’s not my first adventure and I’ve picked up skills and confidence along the way. In particular, I’ve written Bioconductor R packages, dealt with pkgdown/travis issues like #1206, made shiny web applications, analyzed large RNA-seq data, written papers using GoogleDocs, gotten better at asking for help, among other skills.

It's Thurs but here's the "more" I promised https://t.co/4gahvWDbWE @travisci #pkgdown

+my fam of #rstats issues today https://t.co/iGFfHLtasM https://t.co/M8DZEHryJZ https://t.co/FU8qq3f5p6

I'm the pest that visits your GH issues here & there😅Nah, I put lots of 👨🏾‍💻into them!😊 https://t.co/1G0nd5ZwUJ
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) February 28, 2020

I’ve also gotten more comfortable with the idea that I can’t do it all. Others will shortly develop new methods for this type of data, or proper infrastructure to handle this data, or faster visualizations, and so goes on the list. But I’m proud and really happy to say that we built quite the robust prototype. Plus maybe we’ll be involved in shaping this future.

And you noticed that I mentioned we. That’s because I have been learning over the years how to foster collaborations. This particular project involved working with two other members of my workplace who are awesome and that I didn’t know that well. It also involved a new collaboration with someone I’ve known for a while now (we initially met through Twitter in 2014) but hadn’t had the chance to work with. Thus we dove into the unknown together 👩‍🚀🧑‍🚀.

via GIPHY

I feel like we complemented each other quite well and all I can confidently say that our new adventure so far has been very stimulating, even it cost me some sleep.

Woke up at 3:30 am dreaming about code 👨🏻‍💻 😴 and on an off till 5:30. Gave up and went to work early... hopefully I can remember the dreams 😅#CodeWhileDreaming #RestFuelsIdeas pic.twitter.com/h0mYHAaL5W
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) February 19, 2020

Spatial transcriptomics

So, where does spatial transcriptomics come into play and what does it mean? I work with gene activity data which we formally refer to as gene expression 🧬. That is, we measure 🔍🧮 the activity levels of genes for a particular biological condition or tissue sample. For several years now (about since 2007-2009) we have been able to measure many genes from a tissue sample, called bulk RNA-sequencing and abbreviated as RNA-seq.

That’s great! But biology is complicated and a single tissue sample is composed of multiple cells of various types. For example, in the brain there are cells that send signals around (neurons) and others that give structure to the brain. That is why technologies for measuring the gene expression at the single cell level were developed, abbreviated as scRNA-seq. scRNA-seq has been used widely to study mouse brains to live tissue samples.

In recent years I’ve been working with data from the human brain 🧠. The Lieber Institute for Brain Development has about two thousand brain samples. To preserve them for years to come, the brains are frozen 🥶. Cells are a bit fragile and freezing them breaks them. This fact has made it challenging to study data from frozen human brains. Several of my colleagues work on adapting research protocols to handle frozen human brain tissue. The research field overall has been able to generate single nucleus RNA sequencing (snRNA-seq) data and we are all generating some more.

snRNA-seq and scRNA-seq are great because you can measure what genes (pieces of the cell) are active, classify them into groups, and use prior knowledge to label these groups. However, you lose information about what part of the tissue they come from. That’s where technologies for spatial transcriptomics, that is, measuring gene expression 🧬 as close a possible to the single cell level yet retaining spatial coordinates are being actively developed. Thus, you end up with two main sources of data: the gene expression measurements but also images from the tissue (histology information). My coworkers anticipated what could these technologies be used for and what type of research questions they help us answer.

oh hey, that's us! stay tuned for a biorxiv preprint! @LieberInstitute @martinowk @kr_maynard @fellgernon @stephaniehicks @lmwebr https://t.co/uYpwQu52pS
— Andrew Jaffe (@andrewejaffe) February 24, 2020

Our project’s history

My coworkers got early access to a specific new type of spatial transcriptomics technology called Visium from the 10x Genomics company and started piloting it on human brain tissue. They recruited me to their project in early November 2019 (11th) and I recruited more colleagues in early December (4th). Today on February 28th 2020 we made public our research advances, code, and software we built for this project.

How do you keep your documentation concise, friendly, and updated, and stop it from exploding into a giant tangled word mess of multiple files spread across GitHub repos? Asking for a friend.
— Monica Gerber (@mwgerber) February 26, 2020

Given that we have many potential websites others can find us through, we decided to unify as much as possible the documentation even if that meant repeating it. The basic start of our documentation is included further below.

`spatialLIBD`

Welcome to the spatialLIBD project! It is composed of:

a shiny web application that we are hosting at spatial.libd.org/spatialLIBD/ that can handle a limited set of concurrent users,
a Bioconductor package at bioconductor.org/packages/spatialLIBD (or from here) that lets you analyze the data and run a local version of our web application (with our data or yours),
and a research article with the scientific knowledge we drew from this dataset. The analysis code for our project is available here.

The web application allows you to browse the LIBD human dorsolateral pre-frontal cortex (DLPFC) spatial transcriptomics data generated with the 10x Genomics Visium platform. Through the R/Bioconductor package you can also download the data as well as visualize your own datasets using this web application. Please check the bioRxiv pre-print for more details about this project.

If you tweet about this website, the data or the R package please use the #spatialLIBD hashtag. You can find previous tweets that way as shown here. Thank you! Tweet #spatialLIBD

Study design

As a quick overview, the data presented here is from portion of the DLPFC that spans six neuronal layers plus white matter (A) for a total of three subjects with two pairs of spatially adjacent replicates (B). Each dissection of DLPFC was designed to span all six layers plus white matter (C). Using this web application you can explore the expression of known genes such as SNAP25 (D, a neuronal gene), MOBP (E, an oligodendrocyte gene), and known layer markers from mouse studies such as PCP4 (F, a known layer 5 marker gene).

Shiny website mirrors

R/Bioconductor package

The spatialLIBD package contains functions for:

Accessing the spatial transcriptomics data from the LIBD Human Pilot project (code on GitHub) generated with the Visium platform from 10x Genomics. The data is retrieved from Bioconductor’s ExperimentHub.
Visualizing the spot-level spatial gene expression data and clusters.
Inspecting the data interactively either on your computer or through spatial.libd.org/spatialLIBD/.

For more details, please check the documentation website or the Bioconductor package landing page here.

Installation instructions

Get the latest stable R release from CRAN. Then install spatialLIBD from Bioconductor using the following code:

if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")

BiocManager::install("spatialLIBD")

Access the data

Through the spatialLIBD package you can access the processed data in it’s final R format. However, we also provide a table of links so you can download the raw data we received from 10x Genomics.

Processed data

Using spatialLIBD you can access the Human DLPFC spatial transcriptomics data from the 10x Genomics Visium platform. For example, this is the code you can use to access the layer-level data. For more details, check the help file for fetch_data().

## Load the package
library('spatialLIBD')

## Download the spot-level data
sce <- fetch_data(type = 'sce')

## Loading objects:
##   sce

## This is a SingleCellExperiment object
sce

## class: SingleCellExperiment 
## dim: 33538 47681 
## metadata(1): image
## assays(2): counts logcounts
## rownames(33538): ENSG00000243485 ENSG00000237613 ... ENSG00000277475
##   ENSG00000268674
## rowData names(9): source type ... gene_search is_top_hvg
## colnames(47681): AAACAACGAATAGTTC-1 AAACAAGTATCTCCCA-1 ...
##   TTGTTTCCATACAACT-1 TTGTTTGTGTAAATTC-1
## colData names(73): barcode sample_name ... pseudobulk_UMAP_spatial
##   markers_UMAP_spatial
## reducedDimNames(6): PCA TSNE_perplexity50 ... TSNE_perplexity80
##   UMAP_neighbors15
## spikeNames(0):
## altExpNames(0):

## Note the memory size
pryr::object_size(sce)

## 2.08 GB

## Remake the logo image with histology information
sce_image_clus(
    sce = sce,
    clustervar = 'layer_guess_reordered',
    sampleid = '151673',
    colors = libd_layer_colors,
    ... = ' DLPFC Human Brain Layers\nMade with github.com/LieberInstitute/spatialLIBD'
)

Citation

Below is the citation output from using citation('spatialLIBD') in R. Please run this yourself to check for any updates on how to cite spatialLIBD.

citation('spatialLIBD')

## 
## Collado-Torres L, Maynard KR, Jaffe AE (2020). _LIBD Visium spatial
## transcriptomics human pilot data inspector_. doi:
## 10.18129/B9.bioc.spatialLIBD (URL:
## https://doi.org/10.18129/B9.bioc.spatialLIBD),
## https://github.com/LieberInstitute/spatialLIBD - R package version
## 0.99.9, <URL: http://www.bioconductor.org/packages/spatialLIBD>.
## 
## Maynard KR, Collado-Torres L, Weber LM, Uytingco C, Barry BK, Williams
## SR, II JLC, Tran MN, Besich Z, Tippani M, Chew J, Yin Y, Kleinman JE,
## Hyde TM, Rao N, Hicks SC, Martinowich K, Jaffe AE (2020).
## "Transcriptome-scale spatial gene expression in the human dorsolateral
## prefrontal cortex." _bioRxiv_. doi: 10.1101/2020.02.28.969931 (URL:
## https://doi.org/10.1101/2020.02.28.969931), <URL:
## https://www.biorxiv.org/content/10.1101/2020.02.28.969931v1>.
## 
## To see these entries in BibTeX format, use 'print(<citation>,
## bibtex=TRUE)', 'toBibtex(.)', or set
## 'options(citation.bibtex.max=999)'.

Please note that the spatialLIBD was only made possible thanks to many other R and bioinformatics software authors. We have cited their work either in the pre-print or the vignette of the R package.

Closing remarks

Overall, this project has everything that I like: R code, a Bioconductor package, challenging and interest biological data, excellent collaborator team, open communication, and so on.

Now, these are early days of the 10x Genomics Visium platform and there’s much we and others want to learn. So if you have the chance to hear anyone in our team talk more in detail about the project or you simply want to chat with them, here are some opportunities for you to do so as we’d love to collaborate with you or even hire you. Check Stephanie’s tweet and the LIBD career website for more details or simply get in touch with us.

Kristen R Maynard and me will present a The Scientist webinar on March 19th
Keri Martinowich will be at CVCSN 2020 March 26-27th
I’ll present a seminar at LIIGH-UNAM on March 30th
Kristen R Maynard will be at the 2020 Single Cell Symposium on April 20th
Likely Andrew E Jaffe and others will be at The Biology of Genomes 2020 May 5-9th
Stephanie Hicks will present at eRum 2020 May 27-30
Likely some of us will attend BioC2020 July 29-31

Finally, here’s the pre-print twitter thread:

🔥off the press! 👀 our @biorxivpreprint on human 🧠brain @LieberInstitute spatial 🌌🔬transcriptomics data 🧬using Visium @10xGenomics🎉#spatialLIBD

🔍https://t.co/RTW0VscUKR
👩🏾‍💻https://t.co/bsg04XKONr
📚https://t.co/FJDOOzrAJ6
📦https://t.co/Au5jwADGhY https://t.co/PiWEDN9q2N pic.twitter.com/aWy0yLlR50
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) February 29, 2020

Thank you for getting this far!

via GIPHY

References

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.2      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-02-29                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package                * version  date       lib source                                      
##  AnnotationDbi            1.48.0   2019-10-29 [1] Bioconductor                                
##  AnnotationHub            2.18.0   2019-10-29 [1] Bioconductor                                
##  assertthat               0.2.1    2019-03-21 [1] CRAN (R 3.6.0)                              
##  attempt                  0.3.0    2019-04-08 [1] CRAN (R 3.6.0)                              
##  backports                1.1.5    2019-10-02 [1] CRAN (R 3.6.0)                              
##  beeswarm                 0.2.3    2016-04-25 [1] CRAN (R 3.6.0)                              
##  bibtex                   0.4.2.2  2020-01-02 [1] CRAN (R 3.6.0)                              
##  Biobase                * 2.46.0   2019-10-29 [1] Bioconductor                                
##  BiocFileCache            1.10.2   2019-11-08 [1] Bioconductor                                
##  BiocGenerics           * 0.32.0   2019-10-29 [1] Bioconductor                                
##  BiocManager              1.30.10  2019-11-16 [1] CRAN (R 3.6.1)                              
##  BiocNeighbors            1.4.1    2019-11-01 [1] Bioconductor                                
##  BiocParallel           * 1.20.1   2019-12-21 [1] Bioconductor                                
##  BiocSingular             1.2.2    2020-02-14 [1] Bioconductor                                
##  BiocStyle              * 2.14.4   2020-01-09 [1] Bioconductor                                
##  BiocVersion              3.10.1   2019-06-06 [1] Bioconductor                                
##  bit                      1.1-15.2 2020-02-10 [1] CRAN (R 3.6.0)                              
##  bit64                    0.9-7    2017-05-08 [1] CRAN (R 3.6.0)                              
##  bitops                   1.0-6    2013-08-17 [1] CRAN (R 3.6.0)                              
##  blob                     1.2.1    2020-01-20 [1] CRAN (R 3.6.0)                              
##  blogdown                 0.17     2019-11-13 [1] CRAN (R 3.6.1)                              
##  bookdown                 0.17     2020-01-11 [1] CRAN (R 3.6.0)                              
##  cli                      2.0.1    2020-01-08 [1] CRAN (R 3.6.0)                              
##  codetools                0.2-16   2018-12-24 [1] CRAN (R 3.6.2)                              
##  colorout               * 1.2-1    2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)          
##  colorspace               1.4-1    2019-03-18 [1] CRAN (R 3.6.0)                              
##  cowplot                  1.0.0    2019-07-11 [1] CRAN (R 3.6.0)                              
##  crayon                   1.3.4    2017-09-16 [1] CRAN (R 3.6.0)                              
##  curl                     4.3      2019-12-02 [1] CRAN (R 3.6.0)                              
##  data.table               1.12.8   2019-12-09 [1] CRAN (R 3.6.1)                              
##  DBI                      1.1.0    2019-12-15 [1] CRAN (R 3.6.0)                              
##  dbplyr                   1.4.2    2019-06-17 [1] CRAN (R 3.6.0)                              
##  DelayedArray           * 0.12.2   2020-01-06 [1] Bioconductor                                
##  DelayedMatrixStats       1.8.0    2019-10-29 [1] Bioconductor                                
##  desc                     1.2.0    2018-05-01 [1] CRAN (R 3.6.0)                              
##  digest                   0.6.25   2020-02-23 [1] CRAN (R 3.6.0)                              
##  dotCall64                1.0-0    2018-07-30 [1] CRAN (R 3.6.0)                              
##  dplyr                    0.8.4    2020-01-31 [1] CRAN (R 3.6.0)                              
##  DT                       0.12     2020-02-05 [1] CRAN (R 3.6.0)                              
##  evaluate                 0.14     2019-05-28 [1] CRAN (R 3.6.0)                              
##  ExperimentHub            1.12.0   2019-10-29 [1] Bioconductor                                
##  fansi                    0.4.1    2020-01-08 [1] CRAN (R 3.6.0)                              
##  farver                   2.0.3    2020-01-16 [1] CRAN (R 3.6.0)                              
##  fastmap                  1.0.1    2019-10-08 [1] CRAN (R 3.6.0)                              
##  fields                   10.3     2020-02-04 [1] CRAN (R 3.6.0)                              
##  fs                       1.3.1    2019-05-06 [1] CRAN (R 3.6.0)                              
##  GenomeInfoDb           * 1.22.0   2019-10-29 [1] Bioconductor                                
##  GenomeInfoDbData         1.2.2    2019-10-31 [1] Bioconductor                                
##  GenomicRanges          * 1.38.0   2019-10-29 [1] Bioconductor                                
##  ggbeeswarm               0.6.0    2017-08-07 [1] CRAN (R 3.6.0)                              
##  ggplot2                  3.2.1    2019-08-10 [1] CRAN (R 3.6.0)                              
##  glue                     1.3.1    2019-03-12 [1] CRAN (R 3.6.0)                              
##  golem                    0.1      2019-08-05 [1] CRAN (R 3.6.0)                              
##  gridExtra                2.3      2017-09-09 [1] CRAN (R 3.6.0)                              
##  gtable                   0.3.0    2019-03-25 [1] CRAN (R 3.6.0)                              
##  htmltools                0.4.0    2019-10-04 [1] CRAN (R 3.6.0)                              
##  htmlwidgets              1.5.1    2019-10-08 [1] CRAN (R 3.6.0)                              
##  httpuv                   1.5.2    2019-09-11 [1] CRAN (R 3.6.0)                              
##  httr                     1.4.1    2019-08-05 [1] CRAN (R 3.6.0)                              
##  interactiveDisplayBase   1.24.0   2019-10-29 [1] Bioconductor                                
##  IRanges                * 2.20.2   2020-01-13 [1] Bioconductor                                
##  irlba                    2.3.3    2019-02-05 [1] CRAN (R 3.6.0)                              
##  jsonlite                 1.6.1    2020-02-02 [1] CRAN (R 3.6.0)                              
##  knitcitations          * 1.0.10   2019-09-15 [1] CRAN (R 3.6.0)                              
##  knitr                    1.27     2020-01-16 [1] CRAN (R 3.6.0)                              
##  labeling                 0.3      2014-08-23 [1] CRAN (R 3.6.0)                              
##  later                    1.0.0    2019-10-04 [1] CRAN (R 3.6.0)                              
##  lattice                  0.20-38  2018-11-04 [1] CRAN (R 3.6.2)                              
##  lazyeval                 0.2.2    2019-03-15 [1] CRAN (R 3.6.0)                              
##  lifecycle                0.1.0    2019-08-01 [1] CRAN (R 3.6.0)                              
##  lubridate                1.7.4    2018-04-11 [1] CRAN (R 3.6.0)                              
##  magrittr                 1.5      2014-11-22 [1] CRAN (R 3.6.0)                              
##  maps                     3.3.0    2018-04-03 [1] CRAN (R 3.6.0)                              
##  Matrix                   1.2-18   2019-11-27 [1] CRAN (R 3.6.2)                              
##  matrixStats            * 0.55.0   2019-09-07 [1] CRAN (R 3.6.0)                              
##  memoise                  1.1.0    2017-04-21 [1] CRAN (R 3.6.0)                              
##  mime                     0.9      2020-02-04 [1] CRAN (R 3.6.0)                              
##  munsell                  0.5.0    2018-06-12 [1] CRAN (R 3.6.0)                              
##  pillar                   1.4.3    2019-12-20 [1] CRAN (R 3.6.0)                              
##  pkgconfig                2.0.3    2019-09-22 [1] CRAN (R 3.6.1)                              
##  pkgload                  1.0.2    2018-10-29 [1] CRAN (R 3.6.0)                              
##  plotly                   4.9.2    2020-02-12 [1] CRAN (R 3.6.0)                              
##  plyr                     1.8.5    2019-12-10 [1] CRAN (R 3.6.0)                              
##  png                      0.1-7    2013-12-03 [1] CRAN (R 3.6.0)                              
##  Polychrome               1.2.4    2020-02-03 [1] CRAN (R 3.6.0)                              
##  promises                 1.1.0    2019-10-04 [1] CRAN (R 3.6.0)                              
##  pryr                     0.1.4    2018-02-18 [1] CRAN (R 3.6.0)                              
##  purrr                    0.3.3    2019-10-18 [1] CRAN (R 3.6.0)                              
##  R6                       2.4.1    2019-11-12 [1] CRAN (R 3.6.1)                              
##  rappdirs                 0.3.1    2016-03-28 [1] CRAN (R 3.6.0)                              
##  RColorBrewer             1.1-2    2014-12-07 [1] CRAN (R 3.6.0)                              
##  Rcpp                     1.0.3    2019-11-08 [1] CRAN (R 3.6.0)                              
##  RCurl                    1.98-1.1 2020-01-19 [1] CRAN (R 3.6.0)                              
##  RefManageR               1.2.12   2019-04-03 [1] CRAN (R 3.6.0)                              
##  rlang                    0.4.4    2020-01-28 [1] CRAN (R 3.6.0)                              
##  rmarkdown                2.1      2020-01-20 [1] CRAN (R 3.6.0)                              
##  roxygen2                 7.0.2    2019-12-02 [1] CRAN (R 3.6.0)                              
##  rprojroot                1.3-2    2018-01-03 [1] CRAN (R 3.6.0)                              
##  RSQLite                  2.2.0    2020-01-07 [1] CRAN (R 3.6.0)                              
##  rstudioapi               0.11     2020-02-07 [1] CRAN (R 3.6.0)                              
##  rsvd                     1.0.3    2020-02-17 [1] CRAN (R 3.6.0)                              
##  S4Vectors              * 0.24.3   2020-01-18 [1] Bioconductor                                
##  scales                   1.1.0    2019-11-18 [1] CRAN (R 3.6.1)                              
##  scater                   1.14.6   2019-12-16 [1] Bioconductor                                
##  scatterplot3d            0.3-41   2018-03-14 [1] CRAN (R 3.6.0)                              
##  sessioninfo            * 1.1.1    2018-11-05 [1] CRAN (R 3.6.0)                              
##  shiny                    1.4.0    2019-10-10 [1] CRAN (R 3.6.0)                              
##  shinyWidgets             0.5.0    2019-11-18 [1] CRAN (R 3.6.0)                              
##  SingleCellExperiment   * 1.8.0    2019-10-29 [1] Bioconductor                                
##  spam                     2.5-1    2019-12-12 [1] CRAN (R 3.6.0)                              
##  spatialLIBD            * 0.99.9   2020-02-29 [1] Github (LieberInstitute/spatialLIBD@572e2a0)
##  stringi                  1.4.6    2020-02-17 [1] CRAN (R 3.6.0)                              
##  stringr                  1.4.0    2019-02-10 [1] CRAN (R 3.6.0)                              
##  SummarizedExperiment   * 1.16.1   2019-12-19 [1] Bioconductor                                
##  testthat                 2.3.1    2019-12-01 [1] CRAN (R 3.6.0)                              
##  tibble                   2.1.3    2019-06-06 [1] CRAN (R 3.6.0)                              
##  tidyr                    1.0.2    2020-01-24 [1] CRAN (R 3.6.2)                              
##  tidyselect               1.0.0    2020-01-27 [1] CRAN (R 3.6.0)                              
##  usethis                  1.5.1    2019-07-04 [1] CRAN (R 3.6.0)                              
##  vctrs                    0.2.3    2020-02-20 [1] CRAN (R 3.6.0)                              
##  vipor                    0.4.5    2017-03-22 [1] CRAN (R 3.6.0)                              
##  viridis                  0.5.1    2018-03-29 [1] CRAN (R 3.6.0)                              
##  viridisLite              0.3.0    2018-02-01 [1] CRAN (R 3.6.0)                              
##  withr                    2.1.2    2018-03-15 [1] CRAN (R 3.6.0)                              
##  xfun                     0.12     2020-01-13 [1] CRAN (R 3.6.0)                              
##  xml2                     1.2.2    2019-08-09 [1] CRAN (R 3.6.0)                              
##  xtable                   1.8-4    2019-04-21 [1] CRAN (R 3.6.0)                              
##  XVector                  0.26.0   2019-10-29 [1] Bioconductor                                
##  yaml                     2.2.1    2020-02-01 [1] CRAN (R 3.6.0)                              
##  yesno                    0.1.0    2018-04-14 [1] CRAN (R 3.6.0)                              
##  zlibbioc                 1.32.0   2019-10-29 [1] Bioconductor                                
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

Learning from our search history

Wed, 12 Feb 2020 00:00:00 +0000

Origin of the idea

Recently the team I work with has had a few new members and I’ve been thinking lately of ways we could try to help them. The team leader was traveling this week, which gave me the opportunity to come up with a new type of session and test it out. That’s the origin of this learning from our search history idea. We tested it today and I’m quite happy with the results so far, so I thought it would be useful to document what we did and share it with others.

Motivation

The theory

As I show in the slides below, in our group we follow the you must try, and then you must ask framework although with a little different interpretation. The basic goal is to search independently for a period of time (say 15 minutes), but then ask others for help if you are still stuck beyond that point.

In other words, you have to be able to find some answers yourself since that’s part of our job using resources that range from Google Search, to the Bioconductor Support, to the RStudio Community, among other websites. However, we also acknowledge that some questions have difficult answers. Maybe a Stack Overflow thread has multiple answers and not necessarily the top voted question has the answer you are looking for. So instead of spending too much energy, we also tell our members to avoid getting into a rabbit hole for hours. That’s where asking for help comes to play. And you can ask for help from any community you belong to: those involved in the project through Slack, our full team, other scientists in our university, communities we belong to like the rstats community on Twitter, the Bioconductor users community at large, etc.

I did mention that it’s ideal to think about the person who will be helping you answer your question. Small reproducible examples, versions of the software you used, sharing your code on GitHub ¹, the commands you used and the order you used them in can all be valuable for resolving different types of problems. Jenny Bryan has talked much more about this subject, for example in this 2018 webinar.

In practice

Trying for a while then asking for help is all good in practice. However, asking for help is very challenging. It’s scary, you open yourself because you show what you don’t know to other people, and sadly historically many questions have been met with negative feedback on the Internet. Thus, they can make you feel dumb, stupid and many other negative emotions.

I do think that asking for help can be worth it and even wrote a previous blog post on this subject. Some reasons why it’s worth it include being able to move on with your work ², you might learn something new, and if you follow the strategies for helping others help you, you might even figure out the answer yourself.

Now, we all ask for help regardless of how long we have been writing code. Here’s an example tweet that conveys the same message. There are thousands of such tweets online.

I conduct approximately 3948234 programming-related Google searches per day. So does every other developer I know, whether they have 5 weeks of experience or 5 years. Help normalize this practice by tweeting your daily searches with #devgoogle! https://t.co/XfNVUZlhC5
— Lyzi Diamond (@lyzidiamond) January 15, 2019

We are diverse

The team (shown as of May 2018) we work at is very diverse because we all:

have different backgrounds,
have acquired different skills,
are at different career stages (from rotation student up to associate professor),
have different interests,
use different operating systems (from Fedora to Ubuntu to macOS to Windows)
use different tools (mobaxterm vs putty, TextMate vs RStudio, …).

But also because we seek help in different ways ³ and we learn differently.

This means that we have a lot to learn from each other 😊🤓.

Learn from each other exercise

At bit.ly/learnfromsearch you can find a copy of the Google Spreadsheet with the information you need for your team.

Some rules

First, we need to make sure that everyone will feel save to ask questions. That’s why I:

reminded others about our code of conduct,
invited everyone to practice their empathy and be mindful that language matters ⁴,
to keep everyone’s time in their mind as a question could lead to a longer discussion which is best to take another occasion. ⁵

Main steps

The idea is that you pick a problem you solved recently and share:

what you were trying to solve,
the actual text you searched for in Google or elsewhere ⁶,
the link of the website where you find your answer or that guided in this process.

We improved the steps as we were testing this! 🙌🏽🙂

Once everyone has contributed their information to the spreadsheet, we then proceed to go around the table showing and explaining our search use cases.

Goals

Ultimately the goals of this exercise are to

empower ourselves with the knowledge from our teammates,
learn from how we all find help,
build a supportive environment where we feel comfortable asking for help.

Thus in the end, we are enabling our team to fully follow the you must try, and then you must ask framework.

Test session

The first and only session so far went approximately like this:

Min 0-5: get settled in the room.
Min 5-22: presentation about the idea to get members to buy into it.
Min 22-26: demonstration.
Min 26-33: members prepared their information to share with the team.
Min 33-58: members presented their problems, the searches the did, and the solution(s) they found.
Min 58-60: quick wrap up.

My search example

At bit.ly/learnfromsearch I left some examples (anonymized). Mine was about increasing the point size of the output of a plot made with scater::plotReducedDim() which returns a ggplot2 (Wickham, 2016) object. Hence why I searched in Google for ggplot2 increase point size which lead me to the geom_point() reference website. I then tried using + geom_point(size = 20) but that broke the color mapping. I was about to dive into the GitHub code for scater (McCarthy, Campbell, Lun, and Willis, 2017) as this my go-to behavior for many similar quests, but I decided to check the help page using ?scater::plotReducedDim.

… Additional arguments for visualization, see ?“scater-plot-args” for details.

The documentation for ... lead me to ?"scater-plot-args" where I finally found the point_size argument

point_size: Numeric scalar, specifying the size of the points. Defaults to NULL.

and that solved my problem.

So what used to look like this:

library('scater')
library('SummarizedExperiment')

plotReducedDim(
    sce,
    dimred = 'PCA',
    colour_by = 'my_cluster_variable',
    theme_size = 20
)

now looks like this:

plotReducedDim(
    sce,
    dimred = 'PCA',
    colour_by = 'my_cluster_variable',
    theme_size = 20,
    point_size = 5
)

Other examples involve different websites and showcase the diversity of questions we have as a team.

Conclusions

I hope that you like this idea and try it out yourselves. Maybe some of the lessons you learn trying it out could be useful to us as well. Ultimately, the information stored there could be useful for new team members as well as for current members since the spreadsheet becomes like an informal FAQ or team wiki.

via GIPHY

I was strongly encouraged by the feedback two members gave me individually after our trial session. Maybe this is not for everyone as we realized that it’s quite hard to be anonymous while participating ⁷. This idea could evolve into something else, but at least for today, I’m happy with the amount of people that bought into the trial and participated in it. We’ll see what happens next.

Acknowledgments

This blog post was made possible thanks to:

I would also like to acknowledge the general inspiration I’ve gotten from Alison Hill’s educational work. Once the rstudio::conf(2020) videos are available, check the work her intern Desirée de Leon showcased which is related to the following tweet.

Ever seen a giraffe that can fit in a teacup? Time to share the first draft of the R and stats project that Hasse Wallum and I have been working on for some time. More to come! #rstats 🦒 https://t.co/SOlBzop8vz
— Desirée De Leon (@dcossyle) August 14, 2019

P.S. dinámica in Spanish is used for a set of exercises that have a specific purpose in mind. I now realize that dynamic doesn’t hold the same meaning. Oh well 🤷🏾

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] G. Csárdi, R. core, H. Wickham, W. Chang, et al. sessioninfo: R Session Information. R package version 1.1.1. 2018. URL: https://CRAN.R-project.org/package=sessioninfo.

[3] D. J. McCarthy, K. R. Campbell, A. T. L. Lun, and Q. F. Willis. “Scater: pre-processing, quality control, normalisation and visualisation of single-cell RNA-seq data in R”. In: Bioinformatics 33 (8 2017), pp. 1179-1186. DOI: 10.1093/bioinformatics/btw777.

[4] M. Morgan, V. Obenchain, J. Hester, and H. Pagès. SummarizedExperiment: SummarizedExperiment container. R package version 1.16.1. 2019.

[5] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.14.4. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[6] H. Wickham. ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, 2016. ISBN: 978-3-319-24277-4. URL: https://ggplot2.tidyverse.org.

[7] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.2      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-02-12                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package              * version  date       lib source                            
##  assertthat             0.2.1    2019-03-21 [1] CRAN (R 3.6.0)                    
##  beeswarm               0.2.3    2016-04-25 [1] CRAN (R 3.6.0)                    
##  bibtex                 0.4.2.2  2020-01-02 [1] CRAN (R 3.6.0)                    
##  Biobase              * 2.46.0   2019-10-29 [1] Bioconductor                      
##  BiocGenerics         * 0.32.0   2019-10-29 [1] Bioconductor                      
##  BiocManager            1.30.10  2019-11-16 [1] CRAN (R 3.6.1)                    
##  BiocNeighbors          1.4.1    2019-11-01 [1] Bioconductor                      
##  BiocParallel         * 1.20.1   2019-12-21 [1] Bioconductor                      
##  BiocSingular           1.2.1    2019-12-23 [1] Bioconductor                      
##  BiocStyle            * 2.14.4   2020-01-09 [1] Bioconductor                      
##  bitops                 1.0-6    2013-08-17 [1] CRAN (R 3.6.0)                    
##  blogdown               0.17     2019-11-13 [1] CRAN (R 3.6.1)                    
##  bookdown               0.17     2020-01-11 [1] CRAN (R 3.6.0)                    
##  cli                    2.0.1    2020-01-08 [1] CRAN (R 3.6.0)                    
##  colorout             * 1.2-1    2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
##  colorspace             1.4-1    2019-03-18 [1] CRAN (R 3.6.0)                    
##  crayon                 1.3.4    2017-09-16 [1] CRAN (R 3.6.0)                    
##  DelayedArray         * 0.12.2   2020-01-06 [1] Bioconductor                      
##  DelayedMatrixStats     1.8.0    2019-10-29 [1] Bioconductor                      
##  digest                 0.6.23   2019-11-23 [1] CRAN (R 3.6.0)                    
##  dplyr                  0.8.3    2019-07-04 [1] CRAN (R 3.6.0)                    
##  evaluate               0.14     2019-05-28 [1] CRAN (R 3.6.0)                    
##  fansi                  0.4.1    2020-01-08 [1] CRAN (R 3.6.0)                    
##  GenomeInfoDb         * 1.22.0   2019-10-29 [1] Bioconductor                      
##  GenomeInfoDbData       1.2.2    2019-10-31 [1] Bioconductor                      
##  GenomicRanges        * 1.38.0   2019-10-29 [1] Bioconductor                      
##  ggbeeswarm             0.6.0    2017-08-07 [1] CRAN (R 3.6.0)                    
##  ggplot2              * 3.2.1    2019-08-10 [1] CRAN (R 3.6.0)                    
##  glue                   1.3.1    2019-03-12 [1] CRAN (R 3.6.0)                    
##  gridExtra              2.3      2017-09-09 [1] CRAN (R 3.6.0)                    
##  gtable                 0.3.0    2019-03-25 [1] CRAN (R 3.6.0)                    
##  htmltools              0.4.0    2019-10-04 [1] CRAN (R 3.6.0)                    
##  httr                   1.4.1    2019-08-05 [1] CRAN (R 3.6.0)                    
##  IRanges              * 2.20.2   2020-01-13 [1] Bioconductor                      
##  irlba                  2.3.3    2019-02-05 [1] CRAN (R 3.6.0)                    
##  jsonlite               1.6      2018-12-07 [1] CRAN (R 3.6.0)                    
##  knitcitations        * 1.0.10   2019-09-15 [1] CRAN (R 3.6.0)                    
##  knitr                  1.27     2020-01-16 [1] CRAN (R 3.6.0)                    
##  lattice                0.20-38  2018-11-04 [1] CRAN (R 3.6.2)                    
##  lazyeval               0.2.2    2019-03-15 [1] CRAN (R 3.6.0)                    
##  lifecycle              0.1.0    2019-08-01 [1] CRAN (R 3.6.0)                    
##  lubridate              1.7.4    2018-04-11 [1] CRAN (R 3.6.0)                    
##  magrittr               1.5      2014-11-22 [1] CRAN (R 3.6.0)                    
##  Matrix                 1.2-18   2019-11-27 [1] CRAN (R 3.6.2)                    
##  matrixStats          * 0.55.0   2019-09-07 [1] CRAN (R 3.6.0)                    
##  munsell                0.5.0    2018-06-12 [1] CRAN (R 3.6.0)                    
##  pillar                 1.4.3    2019-12-20 [1] CRAN (R 3.6.0)                    
##  pkgconfig              2.0.3    2019-09-22 [1] CRAN (R 3.6.1)                    
##  plyr                   1.8.5    2019-12-10 [1] CRAN (R 3.6.0)                    
##  purrr                  0.3.3    2019-10-18 [1] CRAN (R 3.6.0)                    
##  R6                     2.4.1    2019-11-12 [1] CRAN (R 3.6.1)                    
##  Rcpp                   1.0.3    2019-11-08 [1] CRAN (R 3.6.0)                    
##  RCurl                  1.98-1.1 2020-01-19 [1] CRAN (R 3.6.0)                    
##  RefManageR             1.2.12   2019-04-03 [1] CRAN (R 3.6.0)                    
##  rlang                  0.4.3    2020-01-24 [1] CRAN (R 3.6.2)                    
##  rmarkdown              2.1      2020-01-20 [1] CRAN (R 3.6.0)                    
##  rsvd                   1.0.2    2019-07-29 [1] CRAN (R 3.6.0)                    
##  S4Vectors            * 0.24.3   2020-01-18 [1] Bioconductor                      
##  scales                 1.1.0    2019-11-18 [1] CRAN (R 3.6.1)                    
##  scater               * 1.14.6   2019-12-16 [1] Bioconductor                      
##  sessioninfo          * 1.1.1    2018-11-05 [1] CRAN (R 3.6.0)                    
##  SingleCellExperiment * 1.8.0    2019-10-29 [1] Bioconductor                      
##  stringi                1.4.5    2020-01-11 [1] CRAN (R 3.6.0)                    
##  stringr                1.4.0    2019-02-10 [1] CRAN (R 3.6.0)                    
##  SummarizedExperiment * 1.16.1   2019-12-19 [1] Bioconductor                      
##  tibble                 2.1.3    2019-06-06 [1] CRAN (R 3.6.0)                    
##  tidyselect             0.2.5    2018-10-11 [1] CRAN (R 3.6.0)                    
##  vipor                  0.4.5    2017-03-22 [1] CRAN (R 3.6.0)                    
##  viridis                0.5.1    2018-03-29 [1] CRAN (R 3.6.0)                    
##  viridisLite            0.3.0    2018-02-01 [1] CRAN (R 3.6.0)                    
##  withr                  2.1.2    2018-03-15 [1] CRAN (R 3.6.0)                    
##  xfun                   0.12     2020-01-13 [1] CRAN (R 3.6.0)                    
##  xml2                   1.2.2    2019-08-09 [1] CRAN (R 3.6.0)                    
##  XVector                0.26.0   2019-10-29 [1] Bioconductor                      
##  yaml                   2.2.0    2018-07-25 [1] CRAN (R 3.6.0)                    
##  zlibbioc               1.32.0   2019-10-29 [1] Bioconductor                      
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

So you can link to specific lines and see things you changed through time that might be the source of the problem are among the main reasons why you should try to version control your code.↩︎
Potentially to a more interesting problem than the one you are stuck currently at.↩︎
For example, some use a particular project Slack channel, others the lab one, others through direct messages.↩︎
If you say that something is obvious or easy, you are telling the other person that it should be easy for them too, but we know that it isn’t the case and that’s why they are asking a question.↩︎
If we want to incorporate this exercise into our weekly meetings (maybe once a month), we need to make sure that our team meeting will finish on time.↩︎
The use of keywords can dramatically affect a search results, and this information is useful to convey among ourselves so we can learn to search for help more effectively.↩︎
Basically, you can only be anonymous for those not in the room at the time the question was discussed.↩︎

Conference feelings: from newbie to sponsor

Mon, 03 Feb 2020 00:00:00 +0000

In the summer of 2008, nearly 12 years ago, I attended my first R/Bioconductor conference: BioC2008. Just last week I went to my second rstudio::conf(2020) which I greatly enjoyed. After some tweets exchanges today, I started reflecting on my journey and wanted to share my thoughts.

Why I like going to conferences

I typically enjoy going to conferences, though I also end up exhausted.

via GIPHY

Part of it could be the traveling and all that goes with it, but I think that conferences are mostly mentally taxing. There’s an information deluge which I typically find overwhelming. Sure, I love finding about new work but at some point it’s like collecting bookmarks of stuff to read or try out later ¹. Yet the most exhausting component of going to a conference could be the networking aspect. Meeting others and catching up with friends is extremely rewarding which is why I ultimately sign up for the next conference. How rewarding can it be? My most extreme example is when two professors invited me to apply to the Ph.D. program that I ended up studying. Though simply making friends and having friendly faces around you can go a long way. I’ll expand a bit on this later.

As a newbie

Across my career I have felt as a newbie when attending many different conferences. I might be alone or maybe have a friend at the conference, or it could be my first time at a conference that I haven’t been to. Your mind, like mine, can be filled up with questions that later might seem simple like:

How should I dress?
Is it ok to drink alcohol? ²
How do I not feel alone?
Am I the odd one here?
How can I talk to X or Y person I’ve heard about before but never met?

My answers have likely changed over time and currently are:

Comfortable, but remember that this is a work event. So look professional by getting a haircut and dressing up a bit ³.
Yes, but not too much and if I don’t want any, that’s ok too.
Talk to whoever seems friendly: could be someone new or an old friend or acquaintance.
Maybe, but meh 🤷🏽‍♂️. I do try talking to other latinos or foreigners specially when there’s not many of us around.
This one is still hard, but maybe someone I know knows them and can introduce me. If so, I ask for the favor. Otherwise I try to approach them with a question about their work.

Feeling empowered

Reaching out and trying to meet someone is hard and scary. Something that helped me since BioC2008 was the fact that I had others in mind: I knew that I was going to teach R later that fall at LCG-UNAM, so any questions I could get through or people I could meet were huge wins for me. Thus, thinking about others empowered me to network. And that’s still true for me through our CDSB project at comunidadbioinfo.github.io.

If I’m still feeling shy or alone, I remember a story from my dad: he claims that without approaching others who were having dinner at a table, introducing himself and creating a social connection, his ideas would have not made the Escherichia coli genome paper that helped launch his career. Personally, once I break that barrier and start to feel comfortable, I feel like I can ask anything: specially feedback and support from others on how to advance some ideas of mine. Lately, that’s mostly been related to our CDSB project.

I wish others could feel empowered too, yet it’s hard to get there. For now, I simply try to help others emulating what others have done for me. Ultimately, I know that getting someone to answer the doorbell is the first step towards opening a door 🤗🙌🏾 which motivates me and helps me overcome my shyness. Whatever they do after someone answers the doorbell is up to the person I’m helping. Yet in that moment, the satisfaction of helping others make connections fulfills me. Later on, it brings a smile to my face even if I’m exhausted after the conference.

via GIPHY

Acknowledgments

This blog post was made possible thanks to:

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] G. Csárdi, R. core, H. Wickham, W. Chang, et al. sessioninfo: R Session Information. R package version 1.1.1. 2018. URL: https://CRAN.R-project.org/package=sessioninfo.

[3] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.14.4. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.2      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-02-04                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.6.0)                    
##  bibtex          0.4.2.2 2020-01-02 [1] CRAN (R 3.6.0)                    
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 3.6.1)                    
##  BiocStyle     * 2.14.4  2020-01-09 [1] Bioconductor                      
##  blogdown        0.17    2019-11-13 [1] CRAN (R 3.6.1)                    
##  bookdown        0.17    2020-01-11 [1] CRAN (R 3.6.0)                    
##  cli             2.0.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  colorout      * 1.2-1   2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.6.0)                    
##  digest          0.6.23  2019-11-23 [1] CRAN (R 3.6.0)                    
##  evaluate        0.14    2019-05-28 [1] CRAN (R 3.6.0)                    
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  glue            1.3.1   2019-03-12 [1] CRAN (R 3.6.0)                    
##  htmltools       0.4.0   2019-10-04 [1] CRAN (R 3.6.0)                    
##  httr            1.4.1   2019-08-05 [1] CRAN (R 3.6.0)                    
##  jsonlite        1.6     2018-12-07 [1] CRAN (R 3.6.0)                    
##  knitcitations * 1.0.10  2019-09-15 [1] CRAN (R 3.6.0)                    
##  knitr           1.27    2020-01-16 [1] CRAN (R 3.6.0)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.6.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.6.0)                    
##  plyr            1.8.5   2019-12-10 [1] CRAN (R 3.6.0)                    
##  R6              2.4.1   2019-11-12 [1] CRAN (R 3.6.1)                    
##  Rcpp            1.0.3   2019-11-08 [1] CRAN (R 3.6.0)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.6.0)                    
##  rlang           0.4.3   2020-01-24 [1] CRAN (R 3.6.2)                    
##  rmarkdown       2.1     2020-01-20 [1] CRAN (R 3.6.0)                    
##  sessioninfo   * 1.1.1   2018-11-05 [1] CRAN (R 3.6.0)                    
##  stringi         1.4.5   2020-01-11 [1] CRAN (R 3.6.0)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.6.0)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.6.0)                    
##  xfun            0.12    2020-01-13 [1] CRAN (R 3.6.0)                    
##  xml2            1.2.2   2019-08-09 [1] CRAN (R 3.6.0)                    
##  yaml            2.2.0   2018-07-25 [1] CRAN (R 3.6.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

Which is why I have recently taken to Twitter to take “notes”. Notes that I make public and not necessarily go back and revisit, but if I want to, I know where they are.↩︎
Or not drink alcohol. Whichever you want.↩︎
I don’t wear suits or anything like that though.↩︎

Windows user space issues with installing R packages

Wed, 18 Sep 2019 00:00:00 +0000

Are you a Microsoft Windows R user? Does your Windows username include a space? Like Firstname Lastname. Then you might occassionally run into issues installing packages due to spaces.

Solutions

You could either re-install Windows with a username that has no spaces such as Lastname ¹, but that’s probably not an easy option. Or you can:

Edit your TMP and TEMP environment variables to a location with no spaces, like C:\TEMP following instructions like these ones.
Preferably install R at a location with no spaces, like C:\R, instead of the default C:\Program Files ².

Backstory

A co-worker wanted to install the clusterprofiler Bioconductor package which depends on the DO.db Bioconductor package. This co-worker uses a Windows machine that has a username with a space. Let’s say it was me with Leo Collado to keep them anonymous. The DO.db is only available as a “Source” package with no Windows binary as you can see here.

This means that R has to:

download a tar.gz file,
uncompress it,
and then install it.

In particular, we are talking about DO.db_2.9.tar.gz in this case.

The installation instructions for DO.db are:

if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")

BiocManager::install("DO.db")

Uncompressing

Internally, BiocManager::install() ends up using utils::install.packages(). The first step, downloading, works well. Uncompressing a file in this scenario fails. Why?

> BiocManager::install('DO.db', lib = 'C:/R/R-3.6.0/library')
Bioconductor version 3.9 (BiocManager 1.30.4), R 3.6.0 (2019-04-26)
Installing package(s) 'BiocVersion', 'DO.db'

## removed output

trying URL 'https://bioconductor.org/packages/3.9/data/annotation/src/contrib/DO.db_2.9.tar.gz'
Content type 'application/x-gzip' length 1769978 bytes (1.7 MB)
downloaded 1.7 MB

Error in untar2(tarfile, files, list, exdir, restore_times) :
  incomplete block on file

The downloaded source packages are in
        ‘C:\Users\Leo Collado\AppData\Local\Temp\RtmpqiBJ53\downloaded_packages’

If you search on Google the error message you’ll find links like this one which hint towards an incomplete download. But the download works. You can even download the file and try to run untar() manually and it will fail.

We were told to try installing R at a location with no spaces, so by this point, R was installed at C:\R\R-3.6.0\, hence the lib specification you see above, though it’s irrelevant for these errors.

Uncompressing the tar.gz file is done by utils::untar(). If you look at the code for utils::untar() you’ll see:

## The function definition of utils::untar
function (tarfile, files = NULL, list = FALSE, exdir = ".", compressed = NA, 
  extras = NULL, verbose = FALSE, restore_times = TRUE,
  support_old_tars = Sys.getenv("R_SUPPORT_OLD_TARS", 
    FALSE), tar = Sys.getenv("TAR")) 

## Inside utils::untar()
if (inherits(tarfile, "connection") || identical(tar, "internal")) {
  if (!missing(compressed)) 
    warning("argument 'compressed' is ignored for the internal method")
  return(untar2(tarfile, files, list, exdir, restore_times))
}

## Further below
TAR <- tar
if (!nzchar(TAR) && .Platform$OS.type == "windows" && nzchar(Sys.which("tar.exe"))) 
  TAR <- "tar.exe"
if (!nzchar(TAR) || TAR == "internal") 
  return(untar2(tarfile, files, list, exdir))

In this case, the first untar2() call is called. That is: return(untar2(tarfile, files, list, exdir, restore_times)). The error message incomplete block on file is not really informative in this case because untar2() is not happy when there’s a space in the path to the file ³.

We can get around this untar2() issue by uncompressing the tar.gz file ourselves in a path that has no spaces. For example, if we download DO.db_2.9.tar.gz to C:\R we can uncompress the tar.gz file with:

utils::untar('C:/R/DO.db_2.9.tar.gz')

Installation

Let’s proceed to installing the package.

> install.packages('C:/R/DO.db', repos = NULL, type = 'source', lib = 'C:/R/R-3.6.0/library')
* installing *source* package 'DO.db' ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
ARGUMENT 'Collado\AppData\Local\Temp\Rtmp8EQDjB\Rin2ef05088650f' __ignored__

Error: object 'ÿþ' not found
Execution halted

ERROR: lazy loading failed for package 'DO.db'
* removing 'C:/R/R-3.6.0/library/DO.db'

Warning message:
In install.packages("C:/R/DO.db", repos = NULL, type = "source",  :
  installation of package ‘C:/R/DO.db’ had non-zero exit status
>

Oh noes! It didn’t work 😖 What happened?

If you look closely, you’ll see that it says ARGUMENT 'Collado\AppData\Local\Temp\Rtmp8EQDjB\Rin2ef05088650f' __ignored__. Wait! Collado is the Lastname portion of the username! So we have another space issue ⁴. That structure though looks very familiar, it’s from base::tempdir()!

> tempdir()

[1] "C:\\Users\\Leo Collado\\AppData\\Local\\Temp\\RtmpqiBJ53"

The help file for ?tempdir contained the clues to solving this issue.

By default, tmpdir will be the directory given by tempdir(). This will be a subdirectory of the per-session temporary directory found by the following rule when the R session is started. The environment variables TMPDIR, TMP and TEMP are checked in turn and the first found which points to a writable directory is used: if none succeeds ‘/tmp’ is used. The path should not contain spaces. Note that setting any of these environment variables in the R session has no effect on tempdir(): the per-session temporary directory is created before the interpreter is started.

We can set the TMPDIR environment variable which will be used by the R session spawned inside the installation of DO.db and… it works!

> Sys.setenv(TMPDIR = 'C:/R/tmp_leo')
> Sys.getenv('TMPDIR')
[1] "C:/R/tmp_leo"
>
>
> install.packages('C:/R/DO.db', repos = NULL, type = 'source', lib = 'C:/R/R-3.6.0/library')
* installing *source* package 'DO.db' ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package 'DO.db'
    finding HTML links ... done
    DOANCESTOR                              html 
    DOBASE                                  html 
    DOCHILDREN                              html 
    DOMAPCOUNTS                             html 
    DOOBSOLETE                              html 
    DOOFFSPRING                             html 
    DOPARENTS                               html 
    DOSYNONYM                               html  
    DOTERM                                  html 
    DOTerms-class                           html 
    DOTermsAnnDbBimap                       html 
    DO_dbconn                               html 
** building package indices
** testing if installed package can be loaded from temporary location
*** arch - i386
*** arch - x64
** testing if installed package can be loaded from final location
*** arch - i386
*** arch - x64
** testing if installed package keeps a record of temporary installation path
* DONE (DO.db)
Making 'packages.html' ... done

`clusterProfiler` installation

Now we can continue and install clusterProfiler, right?

> BiocManager::install('clusterProfiler', lib = 'C:/R/R-3.6.0/library')
Bioconductor version 3.9 (BiocManager 1.30.4), R 3.6.0 (2019-04-26)
Installing package(s) 'clusterProfiler'
also installing the dependencies ‘sys’, ‘formatR’, ‘askpass’, ‘farver’, ‘backports’, ‘zeallot’, ‘lambda.r’, ‘futile.options’, ‘curl’, ‘mime’, ‘openssl’, ‘hms’, ‘triebeard’, ‘tweenr’, ‘polyclip’, ‘RcppEigen’, ‘colorspace’, ‘utf8’, ‘vctrs’, ‘futile.logger’, ‘snow’, ‘data.table’, ‘fastmatch’, ‘stringr’, ‘httr’, ‘jsonlite’, ‘progress’, ‘urltools’, ‘xml2’, ‘gridGraphics’, ‘ggforce’, ‘ggrepel’, ‘viridis’, ‘labeling’, ‘munsell’, ‘R6’, ‘cli’, ‘crayon’, ‘fansi’, ‘pillar’, ‘BiocParallel’, ‘fgsea’, ‘reshape2’, ‘cowplot’, ‘europepmc’, ‘ggplotify’, ‘ggraph’, ‘ggridges’, ‘gridExtra’, ‘igraph’, ‘purrr’, ‘RColorBrewer’, ‘UpSetR’, ‘gtable’, ‘lazyeval’, ‘rlang’, ‘scales’, ‘tibble’, ‘viridisLite’, ‘withr’, ‘dplyr’, ‘glue’, ‘stringi’, ‘tidyselect’, ‘DOSE’, ‘enrichplot’, ‘ggplot2’, ‘GO.db’, ‘GOSemSim’, ‘plyr’, ‘qvalue’, ‘rvcheck’, ‘tidyr’

## Delete more output

The downloaded binary packages are in
        C:\Users\Leo Collado\AppData\Local\Temp\RtmpqiBJ53\downloaded_packages
installing the source packages ‘pillar’, ‘GO.db’


trying URL 'https://cloud.r-project.org/src/contrib/pillar_1.4.1.tar.gz'
Content type 'application/x-gzip' length 228572 bytes (223 KB)
downloaded 223 KB

trying URL 'https://bioconductor.org/packages/3.9/data/annotation/src/contrib/GO.db_3.8.2.tar.gz'
Content type 'application/x-gzip' length 31820866 bytes (30.3 MB)
downloaded 30.3 MB

Error in untar2(tarfile, files, list, exdir, restore_times) :
  incomplete block on file

Error in untar2(tarfile, files, list, exdir, restore_times) :
  incomplete block on file

The downloaded source packages are in
        ‘C:\Users\Leo Collado\AppData\Local\Temp\RtmpqiBJ53\downloaded_packages’

The issue is again that utils:::untar2() and thus utils::untar() does not like spaces in the paths. If we look at where the packages were downloaded more closely, we can see a space there at C:\Users\Leo Collado\AppData\Local\Temp\RtmpqiBJ53\downloaded_packages. If you check the help file for utils::install.packages() you’ll see that destdir controls this:

destdir

directory where downloaded packages are stored. If it is NULL (the default) a subdirectory downloaded_packages of the session temporary directory will be used (and the files will be deleted at the end of the session).

If we dig into utils::install.packages() we can see how this comes to play.

## Part of utils::install.packages()
if (is.null(destdir) && nonlocalrepos) {
  tmpd <- file.path(tempdir(), "downloaded_packages")
  if (!file.exists(tmpd) && !dir.create(tmpd)) 
    stop(gettextf("unable to create temporary directory %s", 
      sQuote(tmpd)), domain = NA)
}

Setting the environment variable TMPDIR doesn’t work here as the instructions for tempdir() specify ⁵ although I now see that you can edit the .Renviron file as instructed here.

In any case, if we specify a destdir without spaces we overide the need to control tempdir(), enable utils::untar() to work and we can finally install clusterProfiler 🎉.

> BiocManager::install('clusterProfiler', lib = 'C:/R/R-3.6.0/library', destdir = 'C:/R/dest_leo')
Bioconductor version 3.9 (BiocManager 1.30.4), R 3.6.0 (2019-04-26)
Installing package(s) 'clusterProfiler'
also installing the dependency ‘GO.db’

trying URL 'https://bioconductor.org/packages/3.9/bioc/bin/windows/contrib/3.6/clusterProfiler_3.12.0.zip'
Content type 'application/zip' length 623524 bytes (608 KB)
downloaded 608 KB

package ‘clusterProfiler’ successfully unpacked and MD5 sums checked
installing the source package ‘GO.db’

trying URL 'https://bioconductor.org/packages/3.9/data/annotation/src/contrib/GO.db_3.8.2.tar.gz'
Content type 'application/x-gzip' length 31820866 bytes (30.3 MB)
downloaded 30.3 MB

* installing *source* package 'GO.db' ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package 'GO.db'
    finding HTML links ... done
    GOBASE                                  html 
    GOBPANCESTOR                            html 
    GOBPCHILDREN                            html 
    GOBPOFFSPRING                           html 
    GOBPPARENTS                             html 
    GOCCANCESTOR                            html 
    GOCCCHILDREN                            html 
    GOCCOFFSPRING                           html 
    GOCCPARENTS                             html 
    GOMAPCOUNTS                             html 
    GOMFANCESTOR                            html 
    GOMFCHILDREN                            html 
    GOMFOFFSPRING                           html 
    GOMFPARENTS                             html 
    GOOBSOLETE                              html 
    GOSYNONYM                               html 
    GOTERM                                  html  
    GO_dbconn                               html 
** building package indices
** testing if installed package can be loaded from temporary location
*** arch - i386
*** arch - x64
** testing if installed package can be loaded from final location
*** arch - i386
*** arch - x64
** testing if installed package keeps a record of temporary installation path
* DONE (GO.db)
Making 'packages.html' ... done

Closing

All of the above seemed like too much. In addition, it seemed like BiocManager::install('hypeR', destdir = 'C:/R/dest_leo') was not working ⁶. I likely missed something here earlier today. So controlling utils::tempdir() seemed like the easiest solution such that the defaults of where a package gets downloaded, uncompressed, etc all worked. And the simplest solution we thought of was to create the C:\TEMP directory and update the Windows environment variables TMP and TEMP to point to that location. Then, the rest of the commands worked without having to specify lib or destdir or manually run utils::untar().

As a whole, remember to look for spaces in the error messages! This is specially relevant when you are having issues as a Microsoft Windows R user.

If you have other solutions for Microsoft Windows R users with usernames that have at least one space, please let us know in the comments! Thank you! 🙌🏽

Acknowledgments

This blog post was made possible thanks to:

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] G. Csárdi, R. core, H. Wickham, W. Chang, et al. sessioninfo: R Session Information. R package version 1.1.1. 2018. URL: https://CRAN.R-project.org/package=sessioninfo.

[3] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.14.4. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.2      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-02-12                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.6.0)                    
##  bibtex          0.4.2.2 2020-01-02 [1] CRAN (R 3.6.0)                    
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 3.6.1)                    
##  BiocStyle     * 2.14.4  2020-01-09 [1] Bioconductor                      
##  blogdown        0.17    2019-11-13 [1] CRAN (R 3.6.1)                    
##  bookdown        0.17    2020-01-11 [1] CRAN (R 3.6.0)                    
##  cli             2.0.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  colorout      * 1.2-1   2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.6.0)                    
##  digest          0.6.23  2019-11-23 [1] CRAN (R 3.6.0)                    
##  evaluate        0.14    2019-05-28 [1] CRAN (R 3.6.0)                    
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  glue            1.3.1   2019-03-12 [1] CRAN (R 3.6.0)                    
##  htmltools       0.4.0   2019-10-04 [1] CRAN (R 3.6.0)                    
##  httr            1.4.1   2019-08-05 [1] CRAN (R 3.6.0)                    
##  jsonlite        1.6     2018-12-07 [1] CRAN (R 3.6.0)                    
##  knitcitations * 1.0.10  2019-09-15 [1] CRAN (R 3.6.0)                    
##  knitr           1.27    2020-01-16 [1] CRAN (R 3.6.0)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.6.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.6.0)                    
##  plyr            1.8.5   2019-12-10 [1] CRAN (R 3.6.0)                    
##  R6              2.4.1   2019-11-12 [1] CRAN (R 3.6.1)                    
##  Rcpp            1.0.3   2019-11-08 [1] CRAN (R 3.6.0)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.6.0)                    
##  rlang           0.4.3   2020-01-24 [1] CRAN (R 3.6.2)                    
##  rmarkdown       2.1     2020-01-20 [1] CRAN (R 3.6.0)                    
##  sessioninfo   * 1.1.1   2018-11-05 [1] CRAN (R 3.6.0)                    
##  stringi         1.4.5   2020-01-11 [1] CRAN (R 3.6.0)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.6.0)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.6.0)                    
##  xfun            0.12    2020-01-13 [1] CRAN (R 3.6.0)                    
##  xml2            1.2.2   2019-08-09 [1] CRAN (R 3.6.0)                    
##  yaml            2.2.0   2018-07-25 [1] CRAN (R 3.6.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

This is the case in the Bioconductor Windows build machine where the username is biocbuild as you can see here.↩︎
In the Bioconductor Windows build machines there are again no spaces in the path to the R installation and the library where packages are installed.↩︎
Hopefully in the future Google will lead you to this blog post and you might avoid the rabbit hole I went through!↩︎
By the way, at this point I thought that the error was related to Error: object 'ÿþ' not found and maybe some encoding issues since the DO.db package has Chinese characters.↩︎
Didn’t stop me from trying hehe. I tried using usethis::edit_r_profile() and adding Sys.setenv(TMPDIR = 'C:/R/tmp_leo') but that didn’t work.↩︎
I would need to test this more before reporting it properly to Bioconductor.↩︎

The evolution of my academic career as seen through posters and talks thanks to hugo academic 4.1

Wed, 10 Apr 2019 00:00:00 +0000

The hugo-academic theme which powers my website is active and frequently updated. I don’t update my website that frequently anymore, but I recently found about many of their changes when I made the CDSB website.

We are delighted to share with you our new webpage at https://t.co/rNuiRlNixV with both English and Spanish support

Estamos encantados de compartirles nuestra nueva página web que viene en español e inglés

It'll replace/remplazará a https://t.co/SMtJbOM9KO #rstatsES #diversity pic.twitter.com/1SVg9rcqVf
— ComunidadBioInfo (@CDSBMexico) April 5, 2019

One of the new features that I liked quite a bit was the ability to have landing pages for each person in your team. I wanted to improve my website’s section describing the people I’ve mentored so I decided to update my personal website too. Once I started this process, I realized that talks and publications had drastically changed. You could now have an image per talk or publication, add tags to them and link them to projects. Furthermore, I noticed that I hadn’t uploaded all my posters nor the slides for all my talks. So this whole process of updating my website took quite a bit of time! All this new information is also reflected on my CV now.

The end result: alumni

First of all, I really like how the section describing the people I’ve mentored looks now. You can see the faces of the alumni and navigate to a page describing them in more detail.

For example, check Amy Peterson’s landing page which includes links to all her profiles I know about. I was also able to add some pictures of a key event with her: her MPH capstone final presentation.

My academic career

In this new version of my website you can also quickly take a look at my academic career by browsing through my talks and publications (papers, posters and a book chapter). The papers are fancier, but the posters and the talks tell you a more detailed story of how I’ve advanced through my career so far with in progress talks and posters showing some ideas, many of which we discareded in the end. For example, you can look at the poster I presented at BioC2010 which is when I met Rafa and Ingo from JHU Biostat. Through the posters and talks you can see how the derfinder project came to be and culminated in the publication describing the software, which was my main Ph.D. project. I like how all the talks and posters related to derfinder can easily be found through the project page. The talks now include my Ph.D. defense talk 🙌🏽✌🏽.

You can also see how the templates I used for making talks evolved through time starting with my joint undergrad project with Sur Herrera-Paredes. You can find some presentaitons made with Beamer, others with knitr (before rmarkdown existed), some with RStudio presentations, as well as different PowerPoint templates.

Overall, I’m very happy with my new website and I hope that you enjoy browsing through my academic career. I’ll use it to movitate others by showing them that we all start somewhere and it takes time and effort to grow.

Some code for udpating to hugo academic 4.1.0

If you are updating your hugo-academic website you should be prepared to spend a significant amount of time if you want to include all the details I included. It took me most of my Saturday, Monday afternoon, and several hours on Tuesday to update mine. Here’s some code I used in this process that might be helpful to you.

## I copied my lcolladotorsource/content/talk files to ~/Downloads/ugh-talks
## then I made a new directory named after the initial .md files
## I made this process easier when updating my publications
## (see further below)
p_ori <- '~/Downloads/ugh-talks/'
p_new <- '/users/lcollado/Dropbox/code/lcolladotorsource/content/talk'
f <- dir(p_ori)
ff <- gsub('.md', '', f)


## For testing
# i <- 1

for(i in seq_len(length(f))) {

    f_initial <- file.path(p_ori, f[i])
  f_new <- file.path(p_new, ff[i], 'index.md')
  
  initial <- readLines(f_initial)
  new <- readLines(f_new)
  
  ## For a set of tags present in the previous version of hugo-academic
  ## I was using and the latest one, I found the initial strings
  ## such that I could find and replace the text.
  ## I also made sure to delete that info on the previous version
  ## so I could inspect rapidly if there was any information
  ## on the old version that I hadn't transfered to the new version.
  ## This typically involved some custom urls whose syntax is now
  ## different.
  for(j in c('location', 'abstract ', 'event_url', 'url_video', 'url_slides', 'title', 'url_pdf', 'event ', 'math')) {
    ## for debugging:
    # print(j)
    patt <- paste0('^', j)
    cont <- initial[grep(patt, initial)]
    initial[grep(patt, initial)] <- ''
    stopifnot(length(grep(patt, new)) > 0)
    # print(length(cont))
    new[grep(patt, new)] <- cont
  }
  
  ## For times I had to do this manually
  ## since the names for these tags changed
  j <- 'time_start'
  patt <- paste0('^', j)
  cont <- initial[grep(patt, initial)]
  initial[grep(patt, initial)] <- ''
  new[grep('^date ', new)] <- gsub('time_start', 'date', cont)
  
  ## and the end time wasn't always there
  cont2 <- initial[grep('^time_end', initial)]
  cont2 <- gsub('time_end', 'date_end', cont2)
  if(length(cont2) == 0) cont2 <- gsub('time_start', 'date_end', cont) else initial[grep('^time_end', initial)] <- ""
  new[grep('^date_end ', new)] <- cont2
  
  ## Similarly, the short version of the abstract now has
  ## a different name
  j <- 'abstract_short'
  patt <- paste0('^', j)
  cont <- initial[grep(patt, initial)]
  initial[grep(patt, initial)] <- ''
  new[grep('^summary', new)] <- gsub('abstract_short', 'summary', cont)
  
  ## Replace the old and new files with the updated versions ^^
  writeLines(initial, f_initial)
  writeLines(new, f_new)
}




### Publications

## Again, I moved the original files elsewhere
## and I copied the exampleSite/content/publication/clothing-search
## directory

## I then edited the clothing-search/index.md file
## a little bit before copying it to all the new folders below

p_ori <- '~/Downloads/ugh/'
p_new <- '/users/lcollado/Dropbox/code/lcolladotorsource/content/publication'
f <- dir(p_ori)
ff <- gsub('.md', '', f)


for(i in seq_len(length(f))) {
  
  
  ## Create the new directories from R
  ## coping my modified template publication using the
  ## clothing-search example
  system(paste('cp -R', file.path(p_new, 'clothing-search'), file.path(p_new, ff[i])))
  
  
  
  f_initial <- file.path(p_ori, f[i])
  f_new <- file.path(p_new, ff[i], 'index.md')
  
  initial <- readLines(f_initial)
  new <- readLines(f_new)
  
  ## Update tags just like I did for the talks
  for(j in c('title ', 'date ', 'math ', 'publication ', 'abstract ', 'url_pdf', 'url_project', 'url_dataset', 'url_video', 'url_slides', 'url_code', 'authors ')) {
    #print(j)
    patt <- paste0('^', j)
    cont <- initial[grep(patt, initial)]
    initial[grep(patt, initial)] <- ''
    stopifnot(length(grep(patt, new)) > 0)
    #print(length(cont))
    new[grep(patt, new)] <- cont
  }
  
  writeLines(initial, f_initial)
  writeLines(new, f_new)

}

## I used imagemagick a few times to create the featured.jpg files
convert eurobioc2010-Bacterial-Collado.pdf[0] featured.jpg

## Other times I created those images by taking screenshots
## and then reducing the size to a max width/height of 1000 pixels
## to reduce the file sizes a bit

Acknowledgments

This blog post was made possible thanks to:

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] G. Csárdi, R. core, H. Wickham, W. Chang, et al. sessioninfo: R Session Information. R package version 1.1.1. 2018. URL: https://CRAN.R-project.org/package=sessioninfo.

[3] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.14.4. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.2      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-02-12                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.6.0)                    
##  bibtex          0.4.2.2 2020-01-02 [1] CRAN (R 3.6.0)                    
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 3.6.1)                    
##  BiocStyle     * 2.14.4  2020-01-09 [1] Bioconductor                      
##  blogdown        0.17    2019-11-13 [1] CRAN (R 3.6.1)                    
##  bookdown        0.17    2020-01-11 [1] CRAN (R 3.6.0)                    
##  cli             2.0.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  colorout      * 1.2-1   2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.6.0)                    
##  digest          0.6.23  2019-11-23 [1] CRAN (R 3.6.0)                    
##  evaluate        0.14    2019-05-28 [1] CRAN (R 3.6.0)                    
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  glue            1.3.1   2019-03-12 [1] CRAN (R 3.6.0)                    
##  htmltools       0.4.0   2019-10-04 [1] CRAN (R 3.6.0)                    
##  httr            1.4.1   2019-08-05 [1] CRAN (R 3.6.0)                    
##  jsonlite        1.6     2018-12-07 [1] CRAN (R 3.6.0)                    
##  knitcitations * 1.0.10  2019-09-15 [1] CRAN (R 3.6.0)                    
##  knitr           1.27    2020-01-16 [1] CRAN (R 3.6.0)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.6.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.6.0)                    
##  plyr            1.8.5   2019-12-10 [1] CRAN (R 3.6.0)                    
##  R6              2.4.1   2019-11-12 [1] CRAN (R 3.6.1)                    
##  Rcpp            1.0.3   2019-11-08 [1] CRAN (R 3.6.0)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.6.0)                    
##  rlang           0.4.3   2020-01-24 [1] CRAN (R 3.6.2)                    
##  rmarkdown       2.1     2020-01-20 [1] CRAN (R 3.6.0)                    
##  sessioninfo   * 1.1.1   2018-11-05 [1] CRAN (R 3.6.0)                    
##  stringi         1.4.5   2020-01-11 [1] CRAN (R 3.6.0)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.6.0)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.6.0)                    
##  xfun            0.12    2020-01-13 [1] CRAN (R 3.6.0)                    
##  xml2            1.2.2   2019-08-09 [1] CRAN (R 3.6.0)                    
##  yaml            2.2.0   2018-07-25 [1] CRAN (R 3.6.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

CDSBMexico: remember to apply for BioC2019 travel scholarships

Thu, 28 Feb 2019 00:00:00 +0000

This blog post was first published at the CDSBMexico website.

#CDSBMexico: remember to apply for BioC2019 travel scholarships!!

Due date is March 15thhttps://t.co/iegG0qQzwu

Let us help you! Here we give you some ideas 💡We can also give you feedback via Slack ✅#rstats #bioconductor @Bioconductor #bioc2019 #diversity #LatAm #rstatsES pic.twitter.com/EORg8d2Qxj
— ComunidadBioInfo (@CDSBMexico) March 1, 2019

About 10 months ago we announced our plans to start a new community of R/Bioconductor developers in Mexico and Latin America. The National Bioinformatics Node (NNB-UNAM in Spanish) has organized workshops in bioinformatics and R since 2006 and more frequently since 2010. Last year, 2018, there were five simultaneous one week workshops:

Introduction to R and RStudio
Exploratory data analysis (EDA) of biological data with R
EDA of RNA-seq data and differential expression studies
Genome assembly and annotation from high-throughput sequencing data
And our Latin American R/Bioconductor Developers Workshop

Overall it was a success as described in A recap of CDSB 2018: the start of a community. However, we are not done yet! 💪🏽As we have stated several times, our goal is to turn R users into developers and to increase the representation of Mexicans and Latin Americans at other R events.

BioC2019

The one R event that we followed as a guide last year is the Bioconductor yearly meeting, or BioC. This year, BioC2019 will be held during June 24 to 27 at New York City in the United States. If you are an academic the registration fee is 300 USD prior to May 24th. As of today (Feb 28th), a round trip from Mexico City on June 23 to 28th is about 320-410 USD¹. The accommodation options listed start at 118 USD plus tax a night. As you can see, these numbers add up fairly fast which is a limiting factor for scientists from Mexico (and Latin American in general) to attend BioC2019.

Travel scholarships

However, we want to remind you that there are travel scholarships available! And we, CDSB, want to help you! 🙌🏽

All the details about the travel scholarship and different types of submission proposals are available on the BioC2019 website. But let us summarize the process for you.

You have to apply for either a talk, poster or workshop.
You need a professional website, GitHub profile or your CV to be available online.
You need to answer two questions:

How have you participated in the Bioconductor project in the past?
What do you hope to get out of the conference?

For a talk or poster proposal you have a maximum of 300 words while workshop proposals involve filling out a template.

This is quite a bit of work, but the payoff is immense! You will compete for the opportunity to travel basically for free: free registration and hotel, plus $500 for travel expenses. This can be a career changing opportunity as it was the case for L. Collado-Torres (see his keynote slides) and others.

Get started on your scholarship application!

We know that many of us have a tendency to do things at the last minute. However, we hope 🙏🏽 that you start your application materials as soon as you can. Remember that the deadline is March 15th and no exceptions are made.

Website

If you don’t have a professional website, we recommend that you make one following Emily Zabor’s tutorial. This is the tutorial that one of your fellow CDSBMexico 2018 attendees used to create his website: joseaia.github.io. Alternatively, check out the Chromebook Data Science: intro to R course (available for free if you can’t afford it) which includes a lesson that walks you through this process.

Bioconductor project participation

Next, we know that each of you worked at least in one R package during CDSBMexico 2018 such as rGriffin (check the blog post). Maybe you even submitted your package to Bioconductor. At least, you met and interacted with longtime Bioconductor members such as Martin Morgan and Benilton Carvalho. Since then, you might have worked on other R packages, shiny applications, etc. You might have also answered questions on the Bioconductor support website, reported bugs or opened issues in existing packages, joined the developers mailing list, joined the Bioconductor Slack workspace, etc. Ultimately, you could potentially present a poster or give a talk about the package you worked on last summer, but you’ll need to coordinate with your team members and/or lab principal investigator.

Benefitting from BioC2019

As for what you hope to get out of the conference, well, we will remind you that BioC has different workshops that can help you advance your R skills and knowledge. Furthermore, there’ll be scientific talks from excellent bioinformatics researchers. Then, once you return from the conference, you can spread the knowledge you acquired in a variety of ways: implementing it on your work, teaching at your local university or research institute, participating in the Bioconductor community (Slack, the support website, contributing an R package, etc), presenting at a local R meetup, and participating in other forums. You can also benefit personally from attending the conference by networking with other users and developers that will attend it. From it, you might start new collaborations, find about graduate school programs, find potential postdoc advisors, among other opportunities. Hey, you might help us out with future CDSBMexico events!

CDSBMexico hands on help

If you need help directly, please get in touch with us via the CDSBMexico Slack workspace. If you either don’t remember the Slack workspace or you couldn’t attend CDSBMexico 2018, please get in touch with us via Twitter at twitter.com/CDSBMexico.

We understand that not many will be able to attend BioC2019, so keep an eye open for CDSBMexico 2019! We are hoping to announce our plans soon.

Acknowledgments

This blog post was made possible thanks to:

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] G. Csárdi, R. core, H. Wickham, W. Chang, et al. sessioninfo: R Session Information. R package version 1.1.1. 2018. URL: https://CRAN.R-project.org/package=sessioninfo.

[3] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.14.4. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.2      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-02-12                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.6.0)                    
##  bibtex          0.4.2.2 2020-01-02 [1] CRAN (R 3.6.0)                    
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 3.6.1)                    
##  BiocStyle     * 2.14.4  2020-01-09 [1] Bioconductor                      
##  blogdown        0.17    2019-11-13 [1] CRAN (R 3.6.1)                    
##  bookdown        0.17    2020-01-11 [1] CRAN (R 3.6.0)                    
##  cli             2.0.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  colorout      * 1.2-1   2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.6.0)                    
##  digest          0.6.23  2019-11-23 [1] CRAN (R 3.6.0)                    
##  evaluate        0.14    2019-05-28 [1] CRAN (R 3.6.0)                    
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  glue            1.3.1   2019-03-12 [1] CRAN (R 3.6.0)                    
##  htmltools       0.4.0   2019-10-04 [1] CRAN (R 3.6.0)                    
##  httr            1.4.1   2019-08-05 [1] CRAN (R 3.6.0)                    
##  jsonlite        1.6     2018-12-07 [1] CRAN (R 3.6.0)                    
##  knitcitations * 1.0.10  2019-09-15 [1] CRAN (R 3.6.0)                    
##  knitr           1.27    2020-01-16 [1] CRAN (R 3.6.0)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.6.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.6.0)                    
##  plyr            1.8.5   2019-12-10 [1] CRAN (R 3.6.0)                    
##  R6              2.4.1   2019-11-12 [1] CRAN (R 3.6.1)                    
##  Rcpp            1.0.3   2019-11-08 [1] CRAN (R 3.6.0)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.6.0)                    
##  rlang           0.4.3   2020-01-24 [1] CRAN (R 3.6.2)                    
##  rmarkdown       2.1     2020-01-20 [1] CRAN (R 3.6.0)                    
##  sessioninfo   * 1.1.1   2018-11-05 [1] CRAN (R 3.6.0)                    
##  stringi         1.4.5   2020-01-11 [1] CRAN (R 3.6.0)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.6.0)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.6.0)                    
##  xfun            0.12    2020-01-13 [1] CRAN (R 3.6.0)                    
##  xml2            1.2.2   2019-08-09 [1] CRAN (R 3.6.0)                    
##  yaml            2.2.0   2018-07-25 [1] CRAN (R 3.6.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

Using Google flights for the search.↩︎

The power of tapping into your community for support

Sun, 03 Feb 2019 00:00:00 +0000

This week the owner of my favorite Mexican restaurant in Baltimore, Rosalyn Vera, got death and arson¹ threats. I could have been a bystander, but I tapped into my network and asked for help and she has received it. It’s been great to see the power of the community in action.

The backstory

So, I use R and Bioconductor for work and I get to witness the warmth and mostly friendly #rstats community where daily people ask for help and get it. Born from the R community is the RLadies organization that provides a platform for non-cis gendered males to share their knowledge and increase their visibility. In one of the RLadies Baltimore events, I met a Data Journalist from the Baltimore Sun, Christine Zhang who gave an excellent talk on how she uses R for her job: for example, election maps.

As you know, I’m Mexican. To get over my homesickness these past months I’ve been a frequent customer at a small Mexican restaurant in Baltimore called Cocina Luchadoras. I also frequent live shows from a band called Bad Hombres that plays covers of Spanish rock songs from the 80s and 90s, go to Latin karaoke nights organized by Root Raíces, among other things that bring me joy².

Initial threats and reactions

To warm up this week, I went to get some atole de arroz at Cocina Luchadoras twice. The second time was just about an hour after Rosalyn received death and arson threats on Thursday. I could tell that she was scared but she was also incredibly brave. The threats were because she has placed a sign on her restaurant to support the Donald fundraising campaign from Ilegal Mezcal that says “Donald, eres un pendejo” (Donald, you are stupid) . The person making the threats mentioned that immigrants are taking away jobs from US citizens. As a Hispanic customer, I got scared as well and became suspicious of whoever entered the restaurant.

So, what does one do? How would you react to a hate crime threat? Or well, any threat? Call the police. Do you comply with the request to take down the poster? Hire security? Close and move to a different location? I feel like situations like this put can you at odds between following your ideals and being pragmatic. In this case, Rosalyn had to decide whether to stand by her free speech rights or not.

In the United States of America you expect that the police and the institutions that protect the law will help you. Unlike Guatemala in the 1960s when the police/military tried to silence/kill my grandfather due to the health clinic he and my grandmother had where they would treat people from many backgrounds: he closed down, moved to Mexico, and had to be pragmatic and abandon some of his ideals. Or unlike Mexico in recent times, where there’s widespread insecurity due to the war on drugs, corruption, etc. For example, my father’s house was broken into, he got tied down and was punched until a few ribs broke: he called a Mexican military veteran for help.

So, here in the US, you call the police and wait for them to do their job. But you can also decide to ask for help from your community. So I offered Rosalyn my time and network, and she accepted. This was tricky as I told her, because I didn’t want to either give a platform for the person making the threats or to attract other people like him.

I asked Christine for help on whom to talk to at the Baltimore Sun, and she responded immediately. After some emails and knocking on a couple more doors, the Baltimore Sun published an article by Lillian E Reed with contributions by Thalia Juarez and Catherine Rentz. Catherine has published some great work on hate crimes in Maryland and they’ve made an interactive map to browse the data. From there, the word kept spreading and many started to show their support: Baltimore Reddit, local politicians, and many regular Cocina Luchadores customers.

The threats continue

I went again on Saturday and the place was packed and things seemed to be trending on the right direction. The police had called and were supposed to meet with Rosalyn later. Instead, they didn’t show up and the person making the threats called again…

From an allocation of limited resources, I understand that the Baltimore Police has many things to do and many people to keep safe. Catherine Rentz’ story on hate crimes in Maryland from 2016 and 2017 highlights that this type of crime is likely underreported and that there are very few convictions. Maybe this went through the mind of the policeman who did the report on Thursday: “threat phone call? No witnesses? Ok, not enough evidence… move on”.

I had a very hard time processing the no-show by the police though. Talking to others, we discussed how the law enforcement system is overworked (they are reducing the police officer shift length soon) and swamped. But hey, there’s a supportive community. So I asked for help again. “Who knows someone at the Baltimore Police?” That was the question and soon enough one friend replied, and then emailed a lieutenant. By the next morning (Sunday/today), the Baltimore Police chief was looking to call Rosalyn.

In the meantime, we let the Baltimore Sun know that the person had made the threat again. They quickly posted a new story where they report how Rosalyn first made sure to secure more witnesses this time and secondly told the person making the threats that she has only love for him and thanked him for the phone call. I think that she was really smart with both and she does mean it. In a chat with friends, one of them suggested bringing tacos as a peace offering to the person making the threats.

This second story by the Baltimore Sun got shared by Newsweek in their own story. A friend who is in Mexico right now told me he saw it on the top Apple News stories. Today, there was a security guard at Cocina Luchadoras, who smiled and opened the door for anyone coming in. So spreading the word, asking for help, knocking on doors, and the power of the community is very strong. Strong enough to get the police to pay more attention to Rosalyn’s report.

This is incredible! Typically, a #rstats/#RLadies request for help ends up with a friendly reply pointing to an existing R package or a suggestion on what to do. This time, we gamed the system. So yes, asking for help is challenging but can be worth it.

And surely everyone who’s involved in helping in this story can use it to request the help they need. For example, the Baltimore Sun’s hate crime reporting team can ask for more space and resources for their stories. Or the Baltimore and Maryland Police can ask for help from whoever controls their budget³, so they can get more resources to fight the increase in hate crimes, improve the reporting of hate crime threats, and reach out ot the Hispanic community in Baltimore. If they need a letter of support, they can get one from me if they want it.

For the rest of us, it’s a chance to try to live without fear. Fear is a powerful feeling and I’ve seen it in action. I experienced it first hand when everyone I knew stopped going out in Cuernavaca during an intense period of the war on drugs when military were deployed to the streets. It’s also a chance for us Hispanics in Baltimore to feel like we belong here, which is the same challenge many women and non-cis males have and hence the reason why RLadies is awesome.

And drop by to enjoy the awesome food and women art!

If you drop by in Baltimore, you should go visit Cocina Luchadoras! Their food is deliciously awesome. It starts with handmade tortillas. You use tortillas for many things including tacos. A good tortilla can boost the flavor in a taco, and the ones at Cocina Luchadoras are outstanding! Then there’s the filling which can range from pork (al pastor) to different types of vegetables. And while you wait, you get to experience a women’s empowerment museum with all the arts and crafts they have there about how women are really strong! You also get to meet many interesting patrons and make new friends if you wish to do so. Even if you are not in Baltimore, you should follow them on Instagram or Facebook for great art and food pictures (or Twitter, but they don’t post images there, just links to Instagram).

Acknowledgments

Thanks EpiRen!

Thanks Baltimore Sun staff and the Baltimore Police!

Thanks to the @baltimoresun for getting the word out about this hate crime threat against @CocinaLuchadora! 🇲🇽🇺🇸https://t.co/38oY5XDAK8

By @LillianEReed with contributions by @_ThaliaJuarez and @cdrentz

Thanks @christinezhang!

Thanks again @BaltimorePolice for investigating! https://t.co/rvXlHZBWj0
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) February 1, 2019

This blog post was made possible thanks to:

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] G. Csárdi, R. core, H. Wickham, W. Chang, et al. sessioninfo: R Session Information. R package version 1.1.1. 2018. URL: https://CRAN.R-project.org/package=sessioninfo.

[3] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.14.4. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.2      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-02-12                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.6.0)                    
##  bibtex          0.4.2.2 2020-01-02 [1] CRAN (R 3.6.0)                    
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 3.6.1)                    
##  BiocStyle     * 2.14.4  2020-01-09 [1] Bioconductor                      
##  blogdown        0.17    2019-11-13 [1] CRAN (R 3.6.1)                    
##  bookdown        0.17    2020-01-11 [1] CRAN (R 3.6.0)                    
##  cli             2.0.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  colorout      * 1.2-1   2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.6.0)                    
##  digest          0.6.23  2019-11-23 [1] CRAN (R 3.6.0)                    
##  evaluate        0.14    2019-05-28 [1] CRAN (R 3.6.0)                    
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  glue            1.3.1   2019-03-12 [1] CRAN (R 3.6.0)                    
##  htmltools       0.4.0   2019-10-04 [1] CRAN (R 3.6.0)                    
##  httr            1.4.1   2019-08-05 [1] CRAN (R 3.6.0)                    
##  jsonlite        1.6     2018-12-07 [1] CRAN (R 3.6.0)                    
##  knitcitations * 1.0.10  2019-09-15 [1] CRAN (R 3.6.0)                    
##  knitr           1.27    2020-01-16 [1] CRAN (R 3.6.0)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.6.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.6.0)                    
##  plyr            1.8.5   2019-12-10 [1] CRAN (R 3.6.0)                    
##  R6              2.4.1   2019-11-12 [1] CRAN (R 3.6.1)                    
##  Rcpp            1.0.3   2019-11-08 [1] CRAN (R 3.6.0)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.6.0)                    
##  rlang           0.4.3   2020-01-24 [1] CRAN (R 3.6.2)                    
##  rmarkdown       2.1     2020-01-20 [1] CRAN (R 3.6.0)                    
##  sessioninfo   * 1.1.1   2018-11-05 [1] CRAN (R 3.6.0)                    
##  stringi         1.4.5   2020-01-11 [1] CRAN (R 3.6.0)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.6.0)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.6.0)                    
##  xfun            0.12    2020-01-13 [1] CRAN (R 3.6.0)                    
##  xml2            1.2.2   2019-08-09 [1] CRAN (R 3.6.0)                    
##  yaml            2.2.0   2018-07-25 [1] CRAN (R 3.6.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

Threats to burn down here business.↩︎
Yes, thanks to Marie Kondo my clothes are neatly organized.↩︎
The city council? The state congress? Mayor? Governor? I don’t know.↩︎

Asking for help is challenging but is typically worth it

Mon, 12 Nov 2018 00:00:00 +0000

Recently I’ve been thinking on the subject of asking for help. In short, it’s hard to ask for help. It involves admitting to yourself that you can’t solve the problem alone, opening yourself up, hoping that another person will understand you and guide you in the right direction. Thus it can be painful if your request for help is misunderstood, met with criticism or ignored. Regardless of these obstacles, I think that the potential rewards make it worth it.

I mostly encounter the situation of asking for help in two scenarios. One is about work, mostly R programming. The other one is about personal issues. There are plenty more, like spelling, cultural references, word definitions and academic citations.

Thank you!

I’m always learning more English! Last night it was “yule” via @BenLangmead and others#DeepLearning? 🙃 pic.twitter.com/kX9JTnSBAn
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) November 10, 2018

PCR was invented by Kary Mullis in 1984. He won the Nobel Prize in chemistry for it in 1993. real-time PCR (qPCR) was developed about ten years later and originates with this paper: https://t.co/gpIOr3s7PG
— Lior Pachter (@lpachter) November 10, 2018

Programming context

In the context of R and Bioconductor, and programming in general, it can be scary to ask for help in public forums such as a GitHub issue page, the Bioconductor Support website, the RStudio Community website, etc. That’s because you might get a very short reply that can seem unhelpful if it involves terms you are not familiar with (though it might be a precise reply). There are also issues with online communities that I won’t get into. Over the years, I’ve tried to help those who will potentially answer my questions by:

Double checking the documentation¹.
Providing information to reproduce my problem: sometimes doing so makes me realize my mistake.
Trying for a while (>20 min?) to solve the problem by myself which involves googling quite a bit. However, sometimes I’m not aware of the correct search terms.
Avoiding sounding like I’m demanding help, asking for help immediately or requesting the same person for help lots of times in a short time window².

I’ve also tried to highlight my main question, though this can be hard when you are not sure what’s wrong.

When I get a short reply that seems unfriendly, I try to be appreciative of the time this other person took to reply to me. The terms they use in their reply can be important clues on what to search on Google next.

Language matters and I hate it when others make me feel dumb instead of guiding me to the right documentation/website, while others might dislike receiving tons of loose ended questions. Ultimately, this process requires patience on both sides and I highly value efforts by others that try to make it easier for both the person asking for help and the person responding. See for example:

Here are some my recent requests for help with replies by others:

I still make mistakes by not been clear enough sometimes, that’s why I typically say: “please let me know if you need more information” or something along those lines.

Personal context

Asking for help in a programming context is hard because you are leaving your questions public (most of the time): that way you can reach multiple people that can potentially answer your question, and the people answering the question want them to be public with the hope that the answers will help others. Asking for help in personal context is different, since it tends to be in private.

This might just be me, but I’ve had several situations in my life where I’ve opened up and asked a group of individuals for help. For example, I recently started a conversation with several friends about ways to limit alcohol consumption. It was scary at first to bring this topic up, but the friends I talked to about this subject had also been thinking about it and it lead to some great conversations. I have other examples listed in my timeline ³ on occasions where others have helped me with career advice.

One story that I want to share now is about grad school. I was a first year student and I had an accident one Friday in February. I knew immediately that I needed a surgery to repair my shoulder. I already had experience with orthopedic surgeons in Mexico whose quality was widely variable: it took my family years to find one we could trust for my first surgery. In my first year at a new city and country, I didn’t have my family’s network support. So I reached out via email to the faculty that I had met in person with the hopes that one of them knew a good orthopedic surgeon. This was a personal help request, so I didn’t use the Department wide mailing list. The response was excellent, and not only did some faculty help me find an excellent orthopedic surgeon, but also get an appointment with him for Monday. After my appointment on Monday, I replied to the same set of faculty thanking them for their support.

Later that week, one faculty dropped by my office to reprimand me for emailing all the faculty. I assume that this faculty thought I had emailed the whole Department. This situation shocked me and made me feel awful. My mind started racing and I wanted to clarify the situation with the faculty member. It was taking a lot of my internal courage to approach this faculty member and a second faculty member noticed that I wasn’t well. This second faculty member listened to me⁴ and later invited me to have dinner with his family.

A Department staff member also helped me during these weeks and pointed me to the Johns Hopkins Student Assistance Program where I got help (2 or 3 sessions). Getting professional help was great; some things I remember were that they explained to me that I was mourning since injuring my arm was like losing a piece of me (given my injury history) and they also explained potential cultural differences that lead to the situation with the first faculty member. Once I calmed down, I did email the first faculty to clarify what had happened.

This was a rough situation and probably my toughest moment in grad school, or at least among the top ones. I also value it as a learning moment and it fueled me and continues to do so in some ways. For example, it motivated me to organize Cultural Mixer events (see the ad for the third event), with the hope that they would help us understand each other better⁵, as well as creating the Biostat Social mailing list (now defunct but replaced by a Department Slack). I also think that the reactions to my request for help were overall positive. The balance for sure is positive although there was a bit of negative in the mix. I wish others would just get positive reactions with zero negative ones, but there are no guarantees.

Closing thoughts

With this post I’m hoping to invite others to reflect on situations that involve asking for help. I also hope that others break their internal barriers and ask for help when needed. Echoing words of others, asking for help takes courage. Given the potential rewards, I encourage you to ask for help from your family, friends and colleagues depending on the situation. Simply knowing the term(s) for what you are searching for in code or for what you are feeling (for example, imposter syndrome) can help a lot. Also being directed to professionals, like JHSAP in my shoulder story, and the proper documentation manual, like the R admin devel manual, can propel you in the right direction.

Best of luck walking the uphill path of asking for help!

Acknowledgments

This blog post was made possible thanks to:

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] G. Csárdi, R. core, H. Wickham, W. Chang, et al. sessioninfo: R Session Information. R package version 1.1.1. 2018. URL: https://CRAN.R-project.org/package=sessioninfo.

[3] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.14.4. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.2      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-02-12                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.6.0)                    
##  bibtex          0.4.2.2 2020-01-02 [1] CRAN (R 3.6.0)                    
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 3.6.1)                    
##  BiocStyle     * 2.14.4  2020-01-09 [1] Bioconductor                      
##  blogdown        0.17    2019-11-13 [1] CRAN (R 3.6.1)                    
##  bookdown        0.17    2020-01-11 [1] CRAN (R 3.6.0)                    
##  cli             2.0.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  colorout      * 1.2-1   2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.6.0)                    
##  digest          0.6.23  2019-11-23 [1] CRAN (R 3.6.0)                    
##  evaluate        0.14    2019-05-28 [1] CRAN (R 3.6.0)                    
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  glue            1.3.1   2019-03-12 [1] CRAN (R 3.6.0)                    
##  htmltools       0.4.0   2019-10-04 [1] CRAN (R 3.6.0)                    
##  httr            1.4.1   2019-08-05 [1] CRAN (R 3.6.0)                    
##  jsonlite        1.6     2018-12-07 [1] CRAN (R 3.6.0)                    
##  knitcitations * 1.0.10  2019-09-15 [1] CRAN (R 3.6.0)                    
##  knitr           1.27    2020-01-16 [1] CRAN (R 3.6.0)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.6.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.6.0)                    
##  plyr            1.8.5   2019-12-10 [1] CRAN (R 3.6.0)                    
##  R6              2.4.1   2019-11-12 [1] CRAN (R 3.6.1)                    
##  Rcpp            1.0.3   2019-11-08 [1] CRAN (R 3.6.0)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.6.0)                    
##  rlang           0.4.3   2020-01-24 [1] CRAN (R 3.6.2)                    
##  rmarkdown       2.1     2020-01-20 [1] CRAN (R 3.6.0)                    
##  sessioninfo   * 1.1.1   2018-11-05 [1] CRAN (R 3.6.0)                    
##  stringi         1.4.5   2020-01-11 [1] CRAN (R 3.6.0)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.6.0)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.6.0)                    
##  xfun            0.12    2020-01-13 [1] CRAN (R 3.6.0)                    
##  xml2            1.2.2   2019-08-09 [1] CRAN (R 3.6.0)                    
##  yaml            2.2.0   2018-07-25 [1] CRAN (R 3.6.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

Sometimes I simply don’t understand the documentation, or I’m not aware of the best documentation for my problem.↩︎
Doing any of these typically won’t get you anywhere. The person(s) responding to your help requests is(are) also busy!↩︎
Explained in more detail in this post.↩︎
Or tried to because I couldn’t contain the tears.↩︎
Among other great reasons, I’ll write a blog post about them sometime soon.↩︎

A knot of threads: from CSHL to LCG-UNAM to Aldo Barrientos to diversity scholarship opportunities

Tue, 06 Nov 2018 00:00:00 +0000

I can’t tell you how many times I’ve started to write this post in my mind since May 2018. Today I’m finally typing it on the computer. This will be a rather long post that ties in several threads. I’ll talk about Cold Spring Harbor’s Biology of Genomes conference and its relationship to my undergrad in Mexico. I’ll also introduce you to Aldo Barrientos (1987-2011) who was was my undergrad classmate. Then I’ll tell you about myself and how regardless of your situation, privileged or not, you should ask for help and get to know the people around you. Finally, I want to highlight that many other Mexican researchers are doing great work and would love to be invited to talk about their work. Or simply, the opportunity to compete for scholarships and grants.

CSHL and LCG-UNAM

This past May I was selected to give a talk at Biology of Genomes 2018 (BOG) at Cold Spring Harbor Laboratory (CSHL). This was my first time being there and I had plenty of time to reflect before my talk¹. For me, attending BOG was like a dream come true so I was blown away when I was selected to present at BOG.

I first heard about the BOG conference when I was studying my undergrad in Mexico at LCG-UNAM. My class has a history with the conference too. Basically, part of the class was meeting to discuss what optional courses to create for our fourth year, and well, some miscommunication happened among us. This created a rift in the group and was one of my motivations for focusing on communication as well as giving back. The former lead to me being elected as class president and the latter lead to me volunteering to teach R courses later on my undergrad².

As BOG18 went underway, some of my fellow junior presenters started their talks by mentioning why CSHL was special to them. Following their cue, I wrote some notes just before my presentation.

I really wanted to give a shoutout to all my LCG-UNAM classmates. I also thought of Aldo Barrientos and how it could have been him at BOG. This got me nervous, but I really wanted to do it. I also wanted this to be short and professional. You can see how it turned out for yourself³.

Aldo Barrientos

It’s time for me to tell you more about Aldo Barrientos, my LCG-UNAM classmate. I won’t attempt to tell you his whole story since I think that others could write a much more detailed one that myself. I hope they do and if they are made public, I’ll share them from this blog post.

Aldo came from a low income family in Mexico: I believe that his father was a construction worker⁴. He was smart and advanced through school successfully. He took the entry exam for LCG-UNAM like many people did in 2005. Eventually only around 10-15 percent of the applicants were admitted to this undergrad program. The school was free (25 Mexican cents per year), but it involved relocating to a new city. UNAM has some scholarship programs, but they are not enough. Aldo found a way like he always did. For example, he sold candy and juice to his classmates during lunch break. For reasons that I don’t remember, Aldo didn’t complete his undergrad in 2009 but was about to in 2011.

I remember Aldo as a very joyful person who was always smiling and joking with the rest of us. He had the best dancing skills of anyone at my undergrad. He proved them time and time again, for example winning a dance contest at a party in my house and wearing the Pumas UNAM puma costume in our graduation party and dancing with everyone.

Image source

His path was cut abruptly in 2011. He was stabbed multiple times while waiting for someone. We all knew about this. It’s still unclear to me how everything happened in the following months. But I remember being told that he tried seeing a doctor before going back to his home. One day in 2011, we all started to learn about how grave his situation had turned. I didn’t know what to do⁵ and I talked to David Romero, director (dean?) of CCG-UNAM back then. He guided me and others and we were trying to coordinate ways to help Aldo, but by the time we got to his home it was too late. I regret that we weren’t able to help Aldo earlier.

I still have some emails from 2011 when we were talking about doing something to honour him. We never did as a class, though LCG-UNAM did acknowledge him. Maybe one day we will do an event to remember him.

Aldo faced more challenges that myself and other classmates. Yet I believe that he would have kept solving them one by one; he would have gotten his undergrad degree and had he chosen to stay in academia, then present his research findings at conferences like BOG.

Privileged yes, entitled no

Thinking about Aldo made reflect also on who has been around in my own path, who has helped me and how grateful I am to them. Undoubtedly I’ve had an easier path than others including Aldo. That is, I’ve been privileged.

Still, even with my privileged position it has been a hard fought battle to get to where I am now. I think that a mistake others do is that they assume that they are entitled to something given their privilege instead of working for that thing. This can apply to visas and the like where you still have to apply for them in time.

Along my way I have hesitated many times what to do and have sought guidance from others. Also along my way, I’ve tried to get to know others in my surroundings. Now that I look back I realize that many people were at my position or in junior positions have kept working towards their career goals and advanced in them. So, if you can, get to know others and learn from them. They could be the future Full Professors with Tenure, future Senior Software Engineers or other titles you look up to.

Here’s an incomplete version of my timeline (PDF version). I think that it’s pretty hard to list everyone who has helped you along the way. For example, I’m not mentioning my Ph.D. classmates from whom I learned many things. But it will give you a more detailed picture of the amount of people that have helped me.

Invite LCG-UNAM alumni

Like I said at the beginning of my BOG18 talk, I’m not sure if I’m the first LCG-UNAM alumni to present at BOG but I know for sure that I won’t be the last one. That’s because I’m really proud of my classmates and all the great work they are doing. It’s also easy to go beyond my class to other LCG-UNAM alumni and then even go beyond and extend this to Mexican researchers.

Taken from my timeline, here are a few that I mentioned there (links are to their Twitter profiles):

Maria Gutierrez-Arcelus: LCG-UNAM; Ph.D. at University of Geneva Medical School; Postdoc Harvard University
Mariana Gómez-Schiavon: Ph.D. Duke University; postdoc UCSF
Sur Herrera-Paredes: Ph.D. University of North Carolina; postdoc Standford
Fernando Rabanal Mor: Ph.D. Gregor Mendel Institute; postdoc MPI for Developmental Biology
Carlos Vargas: Winter Genomics, MS and Ph.D. Universitat de Valencia
Mariana Reyes: MS and Ph.D. Universitat de Valencia
Atahualpa Castillo: Ph.D. and postdoc University of Bath
Jimena Mónzon (Ph.D. Univeristy of Bath; Oxford’s Parkinson Disease Centre Career Development Fellow)
Alejandra Zayas: Ph.D. IBT-UNAM
Adriana Verenisse Gonzalez Sandoval: Ph.D. University of Bath; postdoc Stanford
Daniela Robles: Ph.D. Wellcome Trust Sanger Institute; Faculty LIIGH-UNAM
Luis Bolaños: Ph.D. CCG-UNAM; postdoc Oregon State University
Rodrigo García: MS and Ph.D. Universitat de Valencia
Renan Escalante-Chong: Ph.D. Harvard postdoc MIT; Immuneering Corporation
Gianella Garcia Hughes: Ph.D. UT Southwestern
Adrian Cantú: LCG-UNAM; MS IBT-UNAM; INMEGEN; Ph.D. San Diego State University
Ana Georgina (Yina) Cobian: MS IBT-UNAM, Ph.D. San Diego State University
Fernando Riveros-McKay Aguilera: Winter Genomics, Ph.D. Wellcome Trust Sanger Institute
Martín Del Castillo Velasco Herrera: Winter Genomics, Ph.D. Wellcome Trust Sanger Institute
Alejandra Medina-Rivera: LCG-UNAM; Ph.D. CCG-UNAM; postdoc SickKids Toronto; Faculty LIIGH-UNAM
Alejandro Reyes: LCG-UNAM; Ph.D. EMBL; postdoc Harvard
Víctor Moreno Mayar: LCG-UNAM; Ph.D. and postdoc Natural History Museum of Denmark
José Reyes: LCG-UNAM, Ph.D. Harvard

For the full list of LCG-UNAM alumni, check http://www.lcg.unam.mx/es/titulados. There are 257 listed as of October 25, 2018.

Everyone has their own story of breaking barriers and overcoming challenges. All of them would love the opportunity to present their work and to compete for funds to travel to conferences and do so.

It is and for example Mexican grants cannot be used for paying memberships so we cannot easily get the discounts.
— Alejandra Medina-Rivera (@AleMedinaRivera) October 17, 2018

To close off, here are some inspiring or thought provoking tweets:

"Along with a healthy dose of luck, the key attributes needed to produce a worthy PhD thesis are a readiness to accept failure; resilience; persistence; ...and a willingness to commit to very hard work — together with curiosity and a passion for research." https://t.co/o62DODM3iZ
— Mariana Gómez-Schiavon (@mgschiavon) October 28, 2018

When you're early-stage in your career, reading stories of people who have been successful is invaluable.

My #rstats hero is @JennyBryan, and she's shared her story in 2 interviews:
-@rOpenSci profile by @kellrstats: https://t.co/bz9Jj4UJqG
-@statschat: https://t.co/UFWfgIg2qa
— We are R-Ladies (@WeAreRLadies) October 24, 2018

I was told by a PhD thesis committee member that if I returned to Mexico after a postdoc I would "fade into mediocrity". However, I did return to my home country, started my own lab, and are now well funded to investigate the genetics and genomics of cancer in Mexican patients. https://t.co/9inVPvYkPj
— Daniela Robles (@daniela_oaks) July 18, 2018

Got a bad news myself today but then I got an AWESOME one from my mom! She got a 9/10 in her chemistry class, her last class to complete her high school!!! I’m so proud of her & how resilient she’s been! My brother & I couldn’t be happier, prouder & dying to hug her! #nevergiveup
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) May 31, 2018

In Mexico, Day of the Dead was just last week (November 2nd) and Aldo’s birthday would have been a few weeks ago on October 27th. It was time to finally write this blog post. I hope that you take something positive from it.

Acknowledgments

This blog post was made possible thanks to:

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] G. Csárdi, R. core, H. Wickham, W. Chang, et al. sessioninfo: R Session Information. R package version 1.1.1. 2018. URL: https://CRAN.R-project.org/package=sessioninfo.

[3] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.14.4. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

Code for the dates:

## R code for making the dates.pdf file
## which I then imported into Adobe Illustrator for making
## the timeline.png
x <- paste0(1987:2018, '-', c(paste0('0', 1:9), 10:12), '-01')

plot(y = seq_len(length(x)), x = as.Date(x), las = 2, ylab = '', xlab = '', xaxt= 'n')
axis(1, at = as.Date(x), labels = months(as.Date(x)))


library('ggplot2')
library('scales')
df <- data.frame(y = 0, date = as.Date(x))


pdf('dates.pdf', height = 40,  useDingbats = FALSE)
ggplot(df, aes(y, date)) + 
  geom_line() + 
  scale_y_date(limits = c(as.Date("1987-07-01"), as.Date("2018-05-12")), 
               breaks = date_breaks('months'),
               date_labels = "%Y-%m", expand=c(0,30)) +   theme(axis.text.y=element_text(angle=-180))
dev.off()

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.2      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-02-12                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.6.0)                    
##  bibtex          0.4.2.2 2020-01-02 [1] CRAN (R 3.6.0)                    
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 3.6.1)                    
##  BiocStyle     * 2.14.4  2020-01-09 [1] Bioconductor                      
##  blogdown        0.17    2019-11-13 [1] CRAN (R 3.6.1)                    
##  bookdown        0.17    2020-01-11 [1] CRAN (R 3.6.0)                    
##  cli             2.0.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  colorout      * 1.2-1   2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
##  colorspace      1.4-1   2019-03-18 [1] CRAN (R 3.6.0)                    
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.6.0)                    
##  digest          0.6.23  2019-11-23 [1] CRAN (R 3.6.0)                    
##  dplyr           0.8.3   2019-07-04 [1] CRAN (R 3.6.0)                    
##  evaluate        0.14    2019-05-28 [1] CRAN (R 3.6.0)                    
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  ggplot2       * 3.2.1   2019-08-10 [1] CRAN (R 3.6.0)                    
##  glue            1.3.1   2019-03-12 [1] CRAN (R 3.6.0)                    
##  gtable          0.3.0   2019-03-25 [1] CRAN (R 3.6.0)                    
##  htmltools       0.4.0   2019-10-04 [1] CRAN (R 3.6.0)                    
##  httr            1.4.1   2019-08-05 [1] CRAN (R 3.6.0)                    
##  jsonlite        1.6     2018-12-07 [1] CRAN (R 3.6.0)                    
##  knitcitations * 1.0.10  2019-09-15 [1] CRAN (R 3.6.0)                    
##  knitr           1.27    2020-01-16 [1] CRAN (R 3.6.0)                    
##  lazyeval        0.2.2   2019-03-15 [1] CRAN (R 3.6.0)                    
##  lifecycle       0.1.0   2019-08-01 [1] CRAN (R 3.6.0)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.6.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.6.0)                    
##  munsell         0.5.0   2018-06-12 [1] CRAN (R 3.6.0)                    
##  pillar          1.4.3   2019-12-20 [1] CRAN (R 3.6.0)                    
##  pkgconfig       2.0.3   2019-09-22 [1] CRAN (R 3.6.1)                    
##  plyr            1.8.5   2019-12-10 [1] CRAN (R 3.6.0)                    
##  purrr           0.3.3   2019-10-18 [1] CRAN (R 3.6.0)                    
##  R6              2.4.1   2019-11-12 [1] CRAN (R 3.6.1)                    
##  Rcpp            1.0.3   2019-11-08 [1] CRAN (R 3.6.0)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.6.0)                    
##  rlang           0.4.3   2020-01-24 [1] CRAN (R 3.6.2)                    
##  rmarkdown       2.1     2020-01-20 [1] CRAN (R 3.6.0)                    
##  scales        * 1.1.0   2019-11-18 [1] CRAN (R 3.6.1)                    
##  sessioninfo   * 1.1.1   2018-11-05 [1] CRAN (R 3.6.0)                    
##  stringi         1.4.5   2020-01-11 [1] CRAN (R 3.6.0)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.6.0)                    
##  tibble          2.1.3   2019-06-06 [1] CRAN (R 3.6.0)                    
##  tidyselect      0.2.5   2018-10-11 [1] CRAN (R 3.6.0)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.6.0)                    
##  xfun            0.12    2020-01-13 [1] CRAN (R 3.6.0)                    
##  xml2            1.2.2   2019-08-09 [1] CRAN (R 3.6.0)                    
##  yaml            2.2.0   2018-07-25 [1] CRAN (R 3.6.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

Reactions to the blog post

My good old friend Leo sharing very interesting experiences of his path through academic life, public university system in #Mexico, and remembering a very special person who taught us a ton of life lessons. https://t.co/nHAiob30jn
— Luis Bolaños (@lbolanos68) November 7, 2018

I'm reading this with tears in my eyes and joy in my heart. I'm so grateful and proud of my friend Leo. I'm so proud of all my classmates! Love you all! Love you Aldo <3 https://t.co/ifOK9ASPmy
— Mariana ReyesPrieto. (@mreyespri) November 7, 2018

Beautiful and inspiring!! Thanks Leo for writing this, for honoring Aldo and being the voice of many (including myself) who feel lucky and grateful for being a LCG alumni! https://t.co/hDyKKThj8B
— paleogenomics (@paleogenomics) November 8, 2018

Mine was the very last one of the conference.↩︎
Which then lead to my current career. See more at my keynote for the “Latin American R/BioConductor Developers Workshop 2018”.↩︎
My scientific talk was about the BrainSeq Phase II project by the Lieber Institute for Brain Development. It’s now available as a bioRxiv pre-print: 426213. It took a while to finish this project, which contributed to my delay in making this blog post.↩︎
Previous versions of my post incorrectly said that he was a farmer.↩︎
I believe that others asked me since I was or had been the class representative.↩︎

Steps for writing a Twitter summary conference blog post

Tue, 23 Oct 2018 00:00:00 +0000

I recently wrote several blog posts with many tweets¹ as a way of taking notes during the American Society of Human Genetics 2018 conference. A few friends asked me how I did this so fast. The process can be summarized into the following main steps.

Save links of tweets you want to highlight in your post.
Use a hugo-powered blog to obtain the code for embedding tweets easily².
Proofread, edit and post.

My steps detailed

Here’s the actual process I used.

Identify the conference Twitter hashtag. In my recent case, that was ASHG18.
Create a Google doc or some other simple text file you can access easily from your phone and laptop.
While in a presentation, check the latest tweets for the given hashtag. ASGH18 example.

Copy the link of any tweet you find interesting. If there are none for the session you are in, consider writing some tweets yourself with the conference hashtag!

Paste the link into your Google doc.

Once the day is over, copy-paste the contents of your Google doc into a text editor that has the ability to find and replace using regular expressions. For example, TextMate.

Find and replace to create the syntax for embedding tweets. I’ll cover more of this in detail in the next section.

Find:

https://twitter.com/.*/status/

and replace by:

\n`r blogdown::shortcode("tweet", "

Then find:

\?s=[0-9]*

and replace by:

")`\n

Our file now looks like this:

Add any comments you wish, format the headings and proofread.
Make it public and tweet about your post with the conference hashtag!

And you are done! 🙌🏽🎉

Quick embedding

Note that my blog is powered by blogdown (Xie, Hill, and Thomas, 2017) which has the function blogdown::shortcode() for embedding tweets. For example, blogdown::shortcode("tweet", "1054380320533950464") generates the hugo syntax code which then hugo renders into the tweet itself.

I just completed my #ASHG18 survey @GeneticsSociety. Took a few minutes and most of my feedback wouldn't stand out. Except that I did use a box to suggest having a webinar with role playing scenarios that exemplify how the code of conduct can be used to diffuse harassment @jsdron pic.twitter.com/AKZdnGCr25
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 22, 2018

As an alternative, we can use Twitter itself by copy-pasting the code for embedding the tweet directly from the Twitter app/website. On the top right side menu on the tweet itself, select Embed Tweet

then copy-paste the HTML code into your post.

Here’s the code:

<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">I just completed my <a href="https://twitter.com/hashtag/ASHG18?src=hash&amp;ref_src=twsrc%5Etfw">#ASHG18</a> survey <a href="https://twitter.com/GeneticsSociety?ref_src=twsrc%5Etfw">@GeneticsSociety</a>. Took a few minutes and most of my feedback wouldn&#39;t stand out. Except that I did use a box to suggest having a webinar with role playing scenarios that exemplify how the code of conduct can be used to diffuse harassment <a href="https://twitter.com/jsdron?ref_src=twsrc%5Etfw">@jsdron</a> <a href="https://t.co/AKZdnGCr25">pic.twitter.com/AKZdnGCr25</a></p>&mdash; 🇲🇽 Dr. Leonardo Collado-Torres (@fellgernon) <a href="https://twitter.com/fellgernon/status/1054380320533950464?ref_src=twsrc%5Etfw">October 22, 2018</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

and here is how it looks:

I just completed my #ASHG18 survey @GeneticsSociety. Took a few minutes and most of my feedback wouldn't stand out. Except that I did use a box to suggest having a webinar with role playing scenarios that exemplify how the code of conduct can be used to diffuse harassment @jsdron pic.twitter.com/AKZdnGCr25
— 🇲🇽 Dr. Leonardo Collado-Torres (@fellgernon) October 22, 2018

There might be other ways of doing this that are specific to your blog platform.

I also imagine that it should be possible to bookmark the tweets on Twitter and then use an R package to extract your bookmarked tweets from the past 24 hours in order to simplify the process. Maybe this can be done with rtweet although I haven’t dived into it.

New to `blogdown`?

Check:

Also, ask around. You might belong to a group or know someone that is willing to host a guest post in their blogdown blog. For example:

Acknowledgments

This blog post was made possible thanks to:

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] G. Csárdi, R. core, H. Wickham, W. Chang, et al. sessioninfo: R Session Information. R package version 1.1.1. 2018. URL: https://CRAN.R-project.org/package=sessioninfo.

[3] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.14.4. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.2      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-02-12                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.6.0)                    
##  bibtex          0.4.2.2 2020-01-02 [1] CRAN (R 3.6.0)                    
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 3.6.1)                    
##  BiocStyle     * 2.14.4  2020-01-09 [1] Bioconductor                      
##  blogdown        0.17    2019-11-13 [1] CRAN (R 3.6.1)                    
##  bookdown        0.17    2020-01-11 [1] CRAN (R 3.6.0)                    
##  cli             2.0.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  colorout      * 1.2-1   2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.6.0)                    
##  digest          0.6.23  2019-11-23 [1] CRAN (R 3.6.0)                    
##  evaluate        0.14    2019-05-28 [1] CRAN (R 3.6.0)                    
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  glue            1.3.1   2019-03-12 [1] CRAN (R 3.6.0)                    
##  htmltools       0.4.0   2019-10-04 [1] CRAN (R 3.6.0)                    
##  httr            1.4.1   2019-08-05 [1] CRAN (R 3.6.0)                    
##  jsonlite        1.6     2018-12-07 [1] CRAN (R 3.6.0)                    
##  knitcitations * 1.0.10  2019-09-15 [1] CRAN (R 3.6.0)                    
##  knitr           1.27    2020-01-16 [1] CRAN (R 3.6.0)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.6.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.6.0)                    
##  plyr            1.8.5   2019-12-10 [1] CRAN (R 3.6.0)                    
##  R6              2.4.1   2019-11-12 [1] CRAN (R 3.6.1)                    
##  Rcpp            1.0.3   2019-11-08 [1] CRAN (R 3.6.0)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.6.0)                    
##  rlang           0.4.3   2020-01-24 [1] CRAN (R 3.6.2)                    
##  rmarkdown       2.1     2020-01-20 [1] CRAN (R 3.6.0)                    
##  sessioninfo   * 1.1.1   2018-11-05 [1] CRAN (R 3.6.0)                    
##  stringi         1.4.5   2020-01-11 [1] CRAN (R 3.6.0)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.6.0)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.6.0)                    
##  xfun            0.12    2020-01-13 [1] CRAN (R 3.6.0)                    
##  xml2            1.2.2   2019-08-09 [1] CRAN (R 3.6.0)                    
##  yaml            2.2.0   2018-07-25 [1] CRAN (R 3.6.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

ASHG18 day 1 post, day 2, day 3, day 4 and day 5.↩︎
Or any other tool that can get you the embedding code, including the Twitter page itself.↩︎

R Gene Regulatory Interaction Formulator For Inquiring Networks

Thu, 13 Sep 2018 00:00:00 +0000

This blog post was written by ME Martinez-Sanchez, S Muñoz, M Carrillo, E Azpeitia, D Rosenblueth and originally posted at the CDSB blog.¹

In this blog post we will describe the package rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) that was one of the projects developed during the TIB2018-BCDW. We hope to continue developing Griffin and rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019). If you have ideas, suggestions or bugs, please contact us via rGriffin GitHub repo.

The problem

Boolean networks allow us to give a mechanistic explanation to how cell types emerge from regulatory networks. However, inferring the regulatory network and its functions is complex problem, as the available information is often incomplete. rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) uses available biological information (regulatory interactions, cell types, mutants) codified as a set of restrictions and returns the Boolean Networks that satisfy that restrictions. This Boolean networks can then be used to study the biological system.

The rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) package is an R connector to Griffin (Gene Regulatory Interaction Formulator For Inquiring Networks), a java library for inference and analysis of Boolean Network models. Griffin takes as inputs biologically meaningful constraints and turns them into a symbolic representation. Using a SAT engine, Griffin explores the Boolean Network search space, finding all satisfying assignments that are compatible with the specified constraints. The rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) package includes a number of functions to interact with the BoolNet package.

A small example

Let us suppose a cell, we know that this cell has three proteins called a, b and c. We know that a activates b and that b and c inhibit each other. We also suspect that b and c may have positive self-regulatory loops. We can add this interactions to the table as “OPU” (optional, positive, unambiguous). This dataframe is the topology of the network.

Source	Target	Interaction
a	b	+
b	c	-
c	b	-
b	b	OPU
c	c	OPU

Suppose we also have some information of what cell types have been observed. For example, there is a cell type that expresses b, but not a or c and an other cell type that expresses c, but not a or b. There might exist a third cell type that has not been fully characterized where we know that the cell expresses no a or c but we have NO information on b. This dataframe is the attractors of the network.

a	b	c
0	1	0
0	0	1
0	*	0

We can then use this information to create a query. rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) can include other types of information like transition between cell type, cycles, transitions between cell types or mutant cell types.

## Install using:
# devtools::install_github('mar-esther23/rgriffin')
## Note that the package depends on rJava

library('rGriffin')

## Loading required package: rJava

genes = c('a','b','c')
inter = data.frame(source=c('a','b','c', 'b','c'), 
                  target=c('b', 'c', 'b', 'b', 'c'), 
                  type=c('+','-','-','OPU','OPU'),
                    stringsAsFactors = F )
q = create.gquery.graph(inter, genes)
attr = data.frame(a=c(0,'*',0), 
                 b=c(0,1,0), 
                 c=c(0,0,1),
                 stringsAsFactors = F )
q = add.gquery.attractors(q, attr)

Then we can use Griffin to find the networks that behave according with our biological information.

nets = run.gquery(q)
nets

## [1] "targets,factors\na,false\nb,((((!a&b)&!c)|((a&!b)&!c))|((a&b)&!c))\nc,(!b&c)\n"                         
## [2] "targets,factors\na,false\nb,(((((!a&b)&!c)|((a&!b)&!c))|((a&b)&!c))|((a&b)&c))\nc,(!b&c)\n"             
## [3] "targets,factors\na,false\nb,((((((!a&b)&!c)|((!a&b)&c))|((a&!b)&!c))|((a&b)&!c))|((a&b)&c))\nc,(!b&c)\n"
## [4] "targets,factors\na,false\nb,((((!a&b)&!c)|((a&b)&!c))|((a&b)&c))\nc,(!b&c)\n"                           
## [5] "targets,factors\na,false\nb,((((((!a&b)&!c)|((a&!b)&!c))|((a&!b)&c))|((a&b)&!c))|((a&b)&c))\nc,(!b&c)\n"

There are multiple options to integrate BoolNet and rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019). The function get.net.topology() can obtain the topology with interaction signs of a BoolNet network. The function attractor2dataframe() can be used to export a BoolNet attractor as a dataframe that rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) can use. The function run.gquery() includes the option return=’BoolNet’, that return the inferred networks as BoolNet networks.

History

The development of Griffin began in 2013 as a PAPIIT (Programa de Apoyo a Proyectos de Investigación e Innovación Tecnológica) project to solve the inference of Boolean Network models for the Arabidopsis thaliana root stem cell niche. It continued in 2015 with support of Conacyt grant 221341.

In January, 2017 we organized a course in C3-UNAM to teach biologist how to use Griffin. We received two main comments: the input format was too complicated and it was uncomfortable to use the output with other packages. After some consideration we decided to create an R wrapper that could export and import BoolNet networks. We selected BoolNet as it has an good documentation and the package BoolFilter had been designed to work with it.

The development of rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) began during the EOBM 2017 in CUIB. For the following year we continued developing rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) as our schedules allowed. There were multiple challenges during the development: defining user-friendly inputs, using Rjava, and structuring the package. In August 2018, we attended the TIB2018-BCDW where we received valuable guidance from Martin Morgan and Benilton S Carvalho. It was during this workshop that the first version of rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) was finished.

Acknowledgments

Here you can find the tweet about the original blog post:

Check our first contributed blog post: R Gene Regulatory Interaction Formulator For Inquiring Networkshttps://t.co/9azbElGZGH

ME Martinez-Sanchez, S Muñoz, M Carrillo, E Azpeitia, D Rosenblueth

Thx @mt_morgan @benilton!#CDSBMexico #TIB2018 #LatAmBioc18 #rstats #TIB #blog pic.twitter.com/TbJv37fkWQ
— ComunidadBioInfo (@CDSBMexico) August 21, 2018

This blog post was made possible thanks to:

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.8. 2017. URL: https://CRAN.R-project.org/package=knitcitations.

[2] M. Martinez-Sanchez, S. Muñoz, M. Carrillo, E. Azpeitia, et al. rGriffin: Gene Regulatory Interaction Formulator For Inquiring Networks. R package version 0.1. 2019.

[3] H. Wickham, J. Hester, and W. Chang. devtools: Tools to Make Developing R Packages Easier. R package version 2.0.2. 2019. URL: https://CRAN.R-project.org/package=devtools.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                                      
##  version  R version 3.5.2 Patched (2019-02-17 r76113)
##  os       macOS Mojave 10.14.4                       
##  system   x86_64, darwin15.6.0                       
##  ui       X11                                        
##  language (EN)                                       
##  collate  en_US.UTF-8                                
##  ctype    en_US.UTF-8                                
##  tz       America/New_York                           
##  date     2019-04-10                                 
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.5.2)                    
##  backports       1.1.3   2018-12-14 [1] CRAN (R 3.5.0)                    
##  bibtex          0.4.2   2017-06-30 [1] CRAN (R 3.5.0)                    
##  BiocManager     1.30.4  2018-11-13 [1] CRAN (R 3.5.0)                    
##  BiocStyle     * 2.10.0  2018-10-30 [1] Bioconductor                      
##  blogdown        0.11.1  2019-04-05 [1] Github (rstudio/blogdown@5179caf) 
##  bookdown        0.9     2018-12-21 [1] CRAN (R 3.5.0)                    
##  callr           3.2.0   2019-03-15 [1] CRAN (R 3.5.2)                    
##  cli             1.1.0   2019-03-19 [1] CRAN (R 3.5.2)                    
##  colorout      * 1.2-0   2019-02-18 [1] Github (jalvesaq/colorout@cc5fbfa)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.5.0)                    
##  desc            1.2.0   2018-05-01 [1] CRAN (R 3.5.0)                    
##  devtools      * 2.0.2   2019-04-08 [1] CRAN (R 3.5.2)                    
##  digest          0.6.18  2018-10-10 [1] CRAN (R 3.5.0)                    
##  evaluate        0.13    2019-02-12 [1] CRAN (R 3.5.2)                    
##  fs              1.2.7   2019-03-19 [1] CRAN (R 3.5.2)                    
##  glue            1.3.1   2019-03-12 [1] CRAN (R 3.5.2)                    
##  htmltools       0.3.6   2017-04-28 [1] CRAN (R 3.5.0)                    
##  httr            1.4.0   2018-12-11 [1] CRAN (R 3.5.0)                    
##  jsonlite        1.6     2018-12-07 [1] CRAN (R 3.5.0)                    
##  knitcitations * 1.0.8   2017-07-04 [1] CRAN (R 3.5.0)                    
##  knitr           1.22    2019-03-08 [1] CRAN (R 3.5.2)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.5.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.5.0)                    
##  memoise         1.1.0   2017-04-21 [1] CRAN (R 3.5.0)                    
##  pkgbuild        1.0.3   2019-03-20 [1] CRAN (R 3.5.2)                    
##  pkgload         1.0.2   2018-10-29 [1] CRAN (R 3.5.0)                    
##  plyr            1.8.4   2016-06-08 [1] CRAN (R 3.5.0)                    
##  prettyunits     1.0.2   2015-07-13 [1] CRAN (R 3.5.0)                    
##  processx        3.3.0   2019-03-10 [1] CRAN (R 3.5.2)                    
##  ps              1.3.0   2018-12-21 [1] CRAN (R 3.5.0)                    
##  R6              2.4.0   2019-02-14 [1] CRAN (R 3.5.2)                    
##  Rcpp            1.0.1   2019-03-17 [1] CRAN (R 3.5.2)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.5.2)                    
##  remotes         2.0.3   2019-04-09 [1] CRAN (R 3.5.2)                    
##  rGriffin      * 0.1     2019-04-10 [1] local                             
##  rJava         * 0.9-11  2019-03-29 [1] CRAN (R 3.5.2)                    
##  rlang           0.3.4   2019-04-07 [1] CRAN (R 3.5.2)                    
##  rmarkdown       1.12    2019-03-14 [1] CRAN (R 3.5.2)                    
##  rprojroot       1.3-2   2018-01-03 [1] CRAN (R 3.5.0)                    
##  sessioninfo     1.1.1   2018-11-05 [1] CRAN (R 3.5.0)                    
##  stringi         1.4.3   2019-03-12 [1] CRAN (R 3.5.2)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.5.2)                    
##  testthat        2.0.1   2018-10-13 [1] CRAN (R 3.5.0)                    
##  usethis       * 1.5.0   2019-04-07 [1] CRAN (R 3.5.2)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.5.0)                    
##  xfun            0.6     2019-04-02 [1] CRAN (R 3.5.2)                    
##  xml2            1.2.0   2018-01-24 [1] CRAN (R 3.5.0)                    
##  yaml            2.2.0   2018-07-25 [1] CRAN (R 3.5.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.5/Resources/library

I’m re-posting their blog post here with their permission so that their post will be shared more widely via R-Bloggers and R Weekly among other sites since the CDSB blog is currently not linked to those aggregators. The original blog post has been edited so that it evaluates the R code and includes all the information needed for reproducibility.↩

Getting ready to attend rOpenSci unconf18 and probably working on tidyverse-like functions for the first time

Fri, 27 Apr 2018 00:00:00 +0000

It’s Friday 7pm and it’s been a long week with ups and downs¹. But I’m enthused as I write this blog post. In less than a month from now I’ll be attending rOpenSci unconf18 and it’ll be my first time at this type of event. Yay!

Building on my streak of good news, I'm delighted to have been selected to attend @rOpenSci #Unconf18 https://t.co/Xe6lojB7TS ^_^ Also, thanks to the https://t.co/o5OwUWEaBD and @LieberInstitute for their support! I'm hoping to relay as much as I can to @LIBDrstats #rstats
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) April 11, 2018

In my self introduction to everyone attending, I mentioned that I don’t use the pipe (%>%) symbol and that I use <- for assignment.

Recently I had my pre-unconf chat with Stefanie Butland (read more about these chats in her great blog post). In my notes to Stefanie before our chat I had mentioned again my lack of R piping experience and we talked about it. As we talked, it became clear that a blog post on related topics would be useful. Sure, I could have asked directly to the other unconf18 participants, but maybe others from the R community in general can chime in or benefit from reading the answers.

Coding style and git

I’m attending unconf18 with an open mind and I think of myself as someone who can be quite flexible (not with my body!) and accommodating. I’m assuming that most participants at unconf18 will use = for assignment. I’m not looking to start any discussions about the different assignment operators. Simply, I am willing to use whatever the majority uses. Just like I did in my first pull request to the blogdown package (issue 263). I was trying to follow Yihui Xie’s coding style to make his life easier and have a clean (or cleaner) git history. From Yihui’s post on this pull request I can see that he liked it.

Keeping this in mind, I think that following the coding style of others will be something I’ll do at unconf18. I haven’t really worked in any R packages with many developers actively working on the package. But I imagine that setting a common coding style will minimize git conflicts, and no one wants those². I don’t know if we’ll all follow some common guidelines at unconf18. I actually imagine that it’ll be project-specific. Why? Well because you can create an R project in RStudio³ and set some defaults for the project such as:

the number of spaces for tab
line ending conversions

We can also set some global RStudio preferences like whether to auto-indent code after paste, length of lines.

Additionally, we can decide whether we’ll use the RStudio “wand” to reformat code.

Maybe all of this is unnecessary, maybe everyone will work in different non-overlapping functions and thus avoid git conflicts. For example, at my job sometimes I write code with = users, but we don’t work on the same lines of the code file. Later on it becomes easy to identify who wrote which line without having to use git blame (awful name, right?).

Tidyverse-like functions

So far, I think that these coding style issues are minor and will be easily dealt with. I think that we’ll all be able to adapt and instead focus on other problems (like whatever the package is trying to solve) and enjoy the experience (network, build trust as Stefanie put it).

My second concern has to do with something I imagine could require more effort: my homework before the unconf. That is, writing tidyverse-like functions. Like I’ve said, I haven’t used the R pipe %>% symbol. I’ve executed some code with it before, seeing it in many blog posts, but I’ve never actually written functions designed to be compatible with this type of logic.

If I help write a function that is not pipe-friendly, then it might not integrate nicely with other functions written by the rest of the team. It would lead to workarounds in the vignette or maybe having someone else re-factor my first function to make it pipe-friendly. Sure, I would learn from observing others make changes to my code. But I want to take advantage as much as I can from my experience at rOpenSci unconf⁴!

Since I don’t really use %>%, I’m unfamiliar with many things related to pipe-friendly (tidyverse-like) functions. For example:

Do you document them differently? Like make a single Rd file for multiple functions. Or do you make an Rd file per function even if the example usage doesn’t involve %>%?
I know that the first argument is important in pipe-friendly functions. But I ignore if the second and other arguments play a role or not.
Do people use the ellipsis (...) argument in pipe-friendly functions? With my derfinder package I ended up a very deep rabbit hole using .... I explained some of the logic in my dots blog post (there are fair criticisms to going deep with ... in the comments).
How do you write unit tests for pipe-friendly functions? Similar to how you write documentation for them, do the unit tests just test one function a time or do they test several at a time (that is the output after using %>%)?

These and other questions could involve time getting familiar with. Time that I could spend now, before unconf18, learning and at least getting a better sense of what to expect. Maybe I’m complicating myself and worrying too much about this. I imagine that the solution will involve a combination of:

Checking some popular tidyverse packages that use %>%. Like the vignettes/README files and examples.
Reading more about this in a book(s): I don’t know which one though.
Playing around a bit as a user with some of these functions. See what error messages pop up: actually I don’t know how users debug a series of functions tied together via %>%.

Wrapping up

I’m not saying everyone should learn about these topics before unconf18. I think that we are all (well, maybe excluding some) worried about not knowing $X$ or $Y$ R/git/travis/testthat/usethis/etc topic before unconf18. And that will part of why it’ll be great to meet everyone in what is known to be an extremely welcoming R conference ^^ (seriously, check all the unconf17 posts!).

## And I'm done proofreading the post. Yay!
Sys.time()

## [1] "2020-02-12 22:33:17 EST"

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, Morgan, and Huber, 2020)
blogdown (Xie, Hill, and Thomas, 2017)
devtools (Wickham, Hester, and Chang, 2019)
knitcitations (Boettiger, 2019)

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.14.4. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[3] H. Wickham, J. Hester, and W. Chang. devtools: Tools to Make Developing R Packages Easier. R package version 2.2.1. 2019. URL: https://CRAN.R-project.org/package=devtools.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.2      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-02-12                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.6.0)                    
##  backports       1.1.5   2019-10-02 [1] CRAN (R 3.6.0)                    
##  bibtex          0.4.2.2 2020-01-02 [1] CRAN (R 3.6.0)                    
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 3.6.1)                    
##  BiocStyle     * 2.14.4  2020-01-09 [1] Bioconductor                      
##  blogdown        0.17    2019-11-13 [1] CRAN (R 3.6.1)                    
##  bookdown        0.17    2020-01-11 [1] CRAN (R 3.6.0)                    
##  callr           3.4.1   2020-01-24 [1] CRAN (R 3.6.2)                    
##  cli             2.0.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  colorout      * 1.2-1   2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.6.0)                    
##  desc            1.2.0   2018-05-01 [1] CRAN (R 3.6.0)                    
##  devtools      * 2.2.1   2019-09-24 [1] CRAN (R 3.6.1)                    
##  digest          0.6.23  2019-11-23 [1] CRAN (R 3.6.0)                    
##  ellipsis        0.3.0   2019-09-20 [1] CRAN (R 3.6.0)                    
##  evaluate        0.14    2019-05-28 [1] CRAN (R 3.6.0)                    
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  fs              1.3.1   2019-05-06 [1] CRAN (R 3.6.0)                    
##  glue            1.3.1   2019-03-12 [1] CRAN (R 3.6.0)                    
##  htmltools       0.4.0   2019-10-04 [1] CRAN (R 3.6.0)                    
##  httr            1.4.1   2019-08-05 [1] CRAN (R 3.6.0)                    
##  jsonlite        1.6     2018-12-07 [1] CRAN (R 3.6.0)                    
##  knitcitations * 1.0.10  2019-09-15 [1] CRAN (R 3.6.0)                    
##  knitr           1.27    2020-01-16 [1] CRAN (R 3.6.0)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.6.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.6.0)                    
##  memoise         1.1.0   2017-04-21 [1] CRAN (R 3.6.0)                    
##  pkgbuild        1.0.6   2019-10-09 [1] CRAN (R 3.6.0)                    
##  pkgload         1.0.2   2018-10-29 [1] CRAN (R 3.6.0)                    
##  plyr            1.8.5   2019-12-10 [1] CRAN (R 3.6.0)                    
##  prettyunits     1.1.1   2020-01-24 [1] CRAN (R 3.6.2)                    
##  processx        3.4.1   2019-07-18 [1] CRAN (R 3.6.0)                    
##  ps              1.3.0   2018-12-21 [1] CRAN (R 3.6.0)                    
##  R6              2.4.1   2019-11-12 [1] CRAN (R 3.6.1)                    
##  Rcpp            1.0.3   2019-11-08 [1] CRAN (R 3.6.0)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.6.0)                    
##  remotes         2.1.0   2019-06-24 [1] CRAN (R 3.6.0)                    
##  rlang           0.4.3   2020-01-24 [1] CRAN (R 3.6.2)                    
##  rmarkdown       2.1     2020-01-20 [1] CRAN (R 3.6.0)                    
##  rprojroot       1.3-2   2018-01-03 [1] CRAN (R 3.6.0)                    
##  sessioninfo     1.1.1   2018-11-05 [1] CRAN (R 3.6.0)                    
##  stringi         1.4.5   2020-01-11 [1] CRAN (R 3.6.0)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.6.0)                    
##  testthat        2.3.1   2019-12-01 [1] CRAN (R 3.6.0)                    
##  usethis       * 1.5.1   2019-07-04 [1] CRAN (R 3.6.0)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.6.0)                    
##  xfun            0.12    2020-01-13 [1] CRAN (R 3.6.0)                    
##  xml2            1.2.2   2019-08-09 [1] CRAN (R 3.6.0)                    
##  yaml            2.2.0   2018-07-25 [1] CRAN (R 3.6.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

Feeling welcomed can be hard… oh well.↩︎
We could talk about git for a long time too. But I hope that I’ll get by with some git push, git pull, and maybe git branch.↩︎
It’s one of the sponsors http://unconf18.ropensci.org/#sponsors and well, probably want most will be using since it has such nice tools for writing R packages.↩︎
Specially since most only attend one unconf, I think. So it’s different from other conferences that you can experience multiple years and network with the group across a longer period of time: that’s what I’ve done with the Bioconductor meetings.↩︎

Latin American R/BioConductor Developers Workshop 2018

Thu, 19 Apr 2018 00:00:00 +0000

Today I’m excited to invite you to attend the Latin American R/BioConductor Developers Workshop 2018! It’ll be held in Cuernavaca, Mexico from July 30th to August 3rd, 2018. You can find the official announcement in the Bioconductor support website. Let me share with you why I’m excited about this workshop.

At BioC2017, Alejandro Reyes and I talked for a while about the low representation of Latin Americans through out the years that either of us have attended the BioC meetings¹. We wanted to see more Latin faces in person but also in the Bioconductor Support Website and among package contributors. We brainstormed for a while and a at dinner with Bioconductor core members we simply ran with our ideas: we brought up the subject and were pleasantly rewarded with support from the Bioconductor team (Alejandro did most of the convincing!). That is, if we organized a conference/workshop, they would help us by sending someone to be one of the instructors.

Previously, Alejandro and I had already discussed why we like the Bioc meetings so much. One of the keys is that the Bioc meetings have always had events for different levels of R users. That is, you can attend the conference each year and learn something new (that’s how I got my career started). So you can progress from learning R, to mastering several R packages, to contributing R packages to Bioconductor, to brainstorming future directions for the Bioconductor project². Additionally, the Bioc meetings have scientific talks in the mornings so that you can learn new scientific and R developments in a single conference. We’ve seen this model adapted in other Bioc events like BiocAsia, the European Bioconductor Meeting, etc. It would be ideal if we got to that point, but we wanted to start smaller with a focus on developing R packages.

Separately, Heladia Salgado, Romualdo Zayas and others from CCG-UNAM³ and NNB-UNAM⁴ have organized the “Talleres Internacionales de Bioinformática”⁵ (TIBs) for years now. Actually, I was a student in 2006 and gave a lecture in the summer 2010 event. At the TIBs they’ve had an introductory R class for a few years now where Selene Fernandez-Valverde and Alejandra Medina-Rivera have participated as instructors. Among the feedback they’ve received is the desire for a more advanced R course.

Alejandro and I talked to Alejandra Medina-Rivera who shared our enthusiasm for Bioconductor. As a PI in Mexico, Alejandra knows very well how sparse are the R/Bioconductor courses that she can send her students to. By the way, Alejandra is the one who first invited me to attend BioC2008!⁶ We quickly started an email thread with Heladia and basically, the pieces started to fall in the right places. Heladia, Delfino Garcia-Alonso, Daniela Ledezma, Laura Gómez, Shirley Alquicira, Thalia Uranga Martínez, among others have been instrumental in organizing the workshop. Heladia, Alejandra and others also secured crucial funding from CCG-UNAM and LIIGH-UNAM. Together we were able to invite and get all the logistics so that Martin Morgan, Heather Turner, Benilton Carvalho, Selene Fernandez-Valverde, María Teresa Ortiz and Alicia Mastretta Yanes could all help us teach part of the workshop and/or present their work.

Briefly, these are some of the reasons why they are amazing:

Martin: Bioconductor project leader.
Heather: extensive experience teaching R and bioinformatics.
Benilton: he helped organize a similar workshop in 2015: LAb Foundation Brasil. Also an academic relative of Alejandro and myself.
Selene: L’Oréal-UNESCO For Women in Science, has taught at TIBs before.
Teresa: RLadiesCDMX plus just look at her awesome GitHub profile!
Alicia: she has taught an intro to bioinformatics and reproducible genetic analyses course, plus check her GitHub profile!

If you check the workshop schedule we have, you’ll notice that we will have several sessions devoted to actually writing R packages. So the course will cover both the theory and the practice side of things. If you attend the workshop, you’ll have plenty of opportunities to network with your peers and to learn from other R developers. You should be super excited now!! ^^ Plus we have a code of conduct (en Español) to ensure that the conference will be welcoming to everyone.

This is also a reminder that you have to keep trying. Back in 2009 or 2010 I had gotten an offer of support from Bioc for sending someone to Cuernavaca, but due to funding circumstances it fell through. Like I said, all the pieces fell in the right places this time. Plus Heladia Salgado has made it all possible!

2018 is looking like a good year for R courses in Latin America. For example, there’s LatinR and Buenos Aires RLadies plus our workshop! Fingers crossed, I’ll be able to attend the workshop in 2019!

Again, I’m excited to invite you to attend the Latin American R/BioConductor Developers Workshop 2018! Regardless of whether you can attend it, help us spread the word:

Bioc Support website: https://support.bioconductor.org/p/108108/
Bioc twitter: https://twitter.com/Bioconductor/status/986943878535303168

#rstats / @Bioconductor Latin American BioC Developers Workshop 2018 https://t.co/eoL4O4V5xT
— Bioconductor (@Bioconductor) April 19, 2018

Tweet by Alejandro Reyes: https://twitter.com/areyesq/status/986943887976656896

If you are a programmer and want to become a @Bioconductor developer, join the Latin American Developers Workshop in Cuernavaca, Mexico, from Jul 30 to Aug 3. More info here: https://t.co/TAcfqYSiM6 #rstats @CDSBMexico @ccg_unam
— Alejandro Reyes (@areyesq) April 19, 2018

Tweet by LCG UNAM: https://twitter.com/lcgunam/status/986691090920230913

Talleres Internacionales de Bioinformática 2018 y Latin American R/Bioconductor Developers Workshop 30jul-3ago @ccg_unam @unammorelos @unam_mx https://t.co/oHWKjbrDrb #RegistroAbierto pic.twitter.com/bdmOREzhuy
— LCG UNAM (@lcgunam) April 18, 2018

CDSBMexico twitter: https://twitter.com/CDSBMexico/status/986672270964871169

Te invitamos a los Talleres Internacionales de Bioinformática 2018https://t.co/dsSekSp4U8 pic.twitter.com/UN1wmOF9SX
— ComunidadBioInfo (@CDSBMexico) April 18, 2018

CDSBMexico facebook: https://www.facebook.com/events/215830399185346/
Website: https://comunidadbioinfo.github.io/post/r-bioconductor-developers-workshop-2018/
Registration form: http://congresos.nnb.unam.mx/TIB2018/registro/ (select “Latin American R/BioConductor Developers Workshop 2018”)

The Bioc team has been great about financially supporting newcomers! I was one of them: just remember to apply before the deadline expires!↩
It doesn’t have to be in that order. For example, anyone attending the Bioc meeting is welcome at the Developer’s Day and to chime in the discussions about the Bioconductor project.↩
Centro de Ciencias Genómicas: Center for Genomic Sciences↩
Nodo Nacional de Bioinformática: National Bioinformatics Node↩
Internacional Bioinformatics Workshops↩
I then taught a few courses at LCG-UNAM, CCG-UNAM and IBT-UNAM from 2008 to 2011 with some slides in Spanish and English. Alejandro was my TA for one of these courses! For a few years since then I would get emails in Spanish asking about some of those courses. You can still find those courses online, but I bet that the material is outdated. In any case, it’s another example of the limited training opportunities.↩

blogdown archetype (template)

Thu, 08 Mar 2018 00:00:00 +0000

In a recent blog post I wrote about having a template for blogdown posts. I wanted to know if it was possible to do this and make my life (and others hopefully) easier for writing new blog posts that are ready to go with the features I frequently re-use.

In my case, I like using BiocStyle (Oleś, Morgan, and Huber, 2020) for functions such as CRANpkg(), Biocpkg() and Githubpkg(). I also like using knitcitations (Boettiger, 2019) for citing with citep() packages or papers; I use citation() and bib_metadata() to get the necessary information, respectively. Furthermore, I prefer devtools::session_info() (Wickham, Hester, and Chang, 2019) over sessionInfo() since it provides information of where you got the package, which becomes specially relevant when using packages from GitHub. Finally, I like thanking the creators of the tools I use, which in this case is blogdown (Xie, Hill, and Thomas, 2017).

I also like reminding myself how to do some common tasks. Basically, the equivalent of the new R Markdown file you get when using RStudio. In my case, I want to remind myself of the YAML options I frequently use (toc, fig height and width) or how to add screenshots.

My first post on this topic is actually rather messy. That’s because at that time:

I didn’t know about hugo archetypes which are template files,
I hadn’t even thought of making the Insert Image addin.

I went down the rabbit hole of archetypes and blogdown, reported an issue resulated to this topic that was in the way of using archetypes for .Rmd posts. After some encouragement by Yihui Xie, I ended up fixing this issues in my first pull request ever to an RStudio package. The PR also added the Archetype option to the New Post RStudio addin (which I used right now ^^).

Creating my blogdown archetype (template)

To make my archetype (template) for blog posts I looked at the GitHub repo for the theme I’m using. It contains an archetypes directory with several files. I just looked at the one called post.md (check it here) and copied it from themes/hugo-academic/archetypes/post.md to archetypes/post.md. Next I added my favorite R code below the last +++ mark. You can check out the final version here. Below I display the version at the time of writing this blog post (I’m using a .txt extension otherwise the embedded gist doesn’t look good, but you want it to end in .md).

I couldn’t get the archetype to respect some of the YAML that blogdown adds, but well, that’s a single copy-paste I have to do now (if I actually decide to use the custom YAML options which are only for .Rmd posts).

I encourage you to make your own blogdown archetype (template). At least it should remind you to include reproducibility information which matters whenever any R code is involved.

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, Morgan, and Huber, 2020)
blogdown (Xie, Hill, and Thomas, 2017)
devtools (Wickham, Hester, and Chang, 2019)
knitcitations (Boettiger, 2019)

Yihui Xie also talked about my first PR in his blog.

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.14.4. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[3] H. Wickham, J. Hester, and W. Chang. devtools: Tools to Make Developing R Packages Easier. R package version 2.2.1. 2019. URL: https://CRAN.R-project.org/package=devtools.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.2      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-02-12                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.6.0)                    
##  backports       1.1.5   2019-10-02 [1] CRAN (R 3.6.0)                    
##  bibtex          0.4.2.2 2020-01-02 [1] CRAN (R 3.6.0)                    
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 3.6.1)                    
##  BiocStyle     * 2.14.4  2020-01-09 [1] Bioconductor                      
##  blogdown        0.17    2019-11-13 [1] CRAN (R 3.6.1)                    
##  bookdown        0.17    2020-01-11 [1] CRAN (R 3.6.0)                    
##  callr           3.4.1   2020-01-24 [1] CRAN (R 3.6.2)                    
##  cli             2.0.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  colorout      * 1.2-1   2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.6.0)                    
##  desc            1.2.0   2018-05-01 [1] CRAN (R 3.6.0)                    
##  devtools      * 2.2.1   2019-09-24 [1] CRAN (R 3.6.1)                    
##  digest          0.6.23  2019-11-23 [1] CRAN (R 3.6.0)                    
##  ellipsis        0.3.0   2019-09-20 [1] CRAN (R 3.6.0)                    
##  evaluate        0.14    2019-05-28 [1] CRAN (R 3.6.0)                    
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  fs              1.3.1   2019-05-06 [1] CRAN (R 3.6.0)                    
##  glue            1.3.1   2019-03-12 [1] CRAN (R 3.6.0)                    
##  htmltools       0.4.0   2019-10-04 [1] CRAN (R 3.6.0)                    
##  httr            1.4.1   2019-08-05 [1] CRAN (R 3.6.0)                    
##  jsonlite        1.6     2018-12-07 [1] CRAN (R 3.6.0)                    
##  knitcitations * 1.0.10  2019-09-15 [1] CRAN (R 3.6.0)                    
##  knitr           1.27    2020-01-16 [1] CRAN (R 3.6.0)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.6.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.6.0)                    
##  memoise         1.1.0   2017-04-21 [1] CRAN (R 3.6.0)                    
##  pkgbuild        1.0.6   2019-10-09 [1] CRAN (R 3.6.0)                    
##  pkgload         1.0.2   2018-10-29 [1] CRAN (R 3.6.0)                    
##  plyr            1.8.5   2019-12-10 [1] CRAN (R 3.6.0)                    
##  prettyunits     1.1.1   2020-01-24 [1] CRAN (R 3.6.2)                    
##  processx        3.4.1   2019-07-18 [1] CRAN (R 3.6.0)                    
##  ps              1.3.0   2018-12-21 [1] CRAN (R 3.6.0)                    
##  R6              2.4.1   2019-11-12 [1] CRAN (R 3.6.1)                    
##  Rcpp            1.0.3   2019-11-08 [1] CRAN (R 3.6.0)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.6.0)                    
##  remotes         2.1.0   2019-06-24 [1] CRAN (R 3.6.0)                    
##  rlang           0.4.3   2020-01-24 [1] CRAN (R 3.6.2)                    
##  rmarkdown       2.1     2020-01-20 [1] CRAN (R 3.6.0)                    
##  rprojroot       1.3-2   2018-01-03 [1] CRAN (R 3.6.0)                    
##  sessioninfo     1.1.1   2018-11-05 [1] CRAN (R 3.6.0)                    
##  stringi         1.4.5   2020-01-11 [1] CRAN (R 3.6.0)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.6.0)                    
##  testthat        2.3.1   2019-12-01 [1] CRAN (R 3.6.0)                    
##  usethis       * 1.5.1   2019-07-04 [1] CRAN (R 3.6.0)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.6.0)                    
##  xfun            0.12    2020-01-13 [1] CRAN (R 3.6.0)                    
##  xml2            1.2.2   2019-08-09 [1] CRAN (R 3.6.0)                    
##  yaml            2.2.0   2018-07-25 [1] CRAN (R 3.6.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

blogdown Insert Image addin

Wed, 07 Mar 2018 00:00:00 +0000

Have you ever tried inserting an image into a blogdown post? Maybe you have, or maybe you tried and gave up. Lets first review the hard way before getting to the solution I contributed.

The hard way

The process involves copying the target image to the static directory that corresponds to the blogdown post. Lets say that your post is called 2018-03-07-my-new-post.Rmd and lives at content/post/, so it’s full path is content/post/2018-03-07-my-new-post.Rmd. When you run the RStudio blogdown addin Serve Site, behind curtains the directory static/post/2018-03-07-my-new-post_files is created and inside it you can find the images made by your R code: likely at static/post/2018-03-07-my-new-post_files/figure_html.

So far everything is working! But now you want to add a screenshot or some other image to your blog post. Lets say that your image is ~/Desktop/screenshot.jpg. Your ~/Desktop directory is not part of your blogdown directory and well, simply put, your website won’t find the image. We need to put it in a location that will be made public by hugo. That is, we need to put it inside static/post/2018-03-07-my-new-post_files (or anywhere inside static, but we like to keep things tidy!).

Ok, so we copy our screenshot file ~/Desktop/screenshot.jpg and save it as static/post/2018-03-07-my-new-post_files/screenshot.jpg. The next time we render our site and publish it, the figure will be available in the web. But it’s still not part of our blogdown post.

So we need to use either the Markdown or HTML syntax for adding the image. Maybe your initial thought is to use:

![](screenshot.jpg)

Except that will not work. We need to use almost all the path (just remove static) as shown below:

![](/post/2018-03-07-my-new-post_files/screenshot.jpg)

If you want to edit the height or width, then you need to use the HTML syntax. Something like:

<img alt = 'my new screenshot' width='200' src='/post/2018-03-07-my-new-post_files/screenshot.jpg' />

hard way notes

You could have also used knitr::include_graphics() and let blogdown copy it to the final location in static and link to it appropriately. However, you would have to keep your original images organized in a way that won’t bother hugo.

Another option that I used for a while, even in the days when my blog was based on Jekyll, is to render the figures yourself and copy the directory with the figures, plus mess around with how they are linked from R. Details here. Not something I recommend doing now.

Insert Image addin: aka, the easy way

If you are using blogdown, you most likely (you should if you can) are using RStudio and the great blogdown addins: New Post and Serve Site. I just recently started using them in the past few days and looking at the code I realized that it should be possible to make an addin that lets you:

select a target image,
copies the target image to the appropriate location under static,
gives you the correct code for linking the image.

Yihui Xie loved the idea (I think it’s fair to say that ^^) and helped me polish it in the pull request that implements it. He then refined the code even more!

Features of the Insert Image addin

The final features, at least as implemented in blogdown version 0.5.7 are:

Select an image from anywhere in your computer.
Automatically generate a candidate final location for your image under static, which you can edit. Useful if you want to rename the final figure.
Allow specifying the alternate description of the image (alt), height and width.
If the target image file exists, a dynamic menu shows up that asks you whether to overwrite it or not.
The final syntax is Markdown unless a width or height are used, in which case it uses HTML code.

Yihui Xie hinted at other possible future features, which maybe you can help implement.

Using the Insert Image addin

Step 1: install appropriate blogdown version

First of all, at the time of writing this post, you need the development version of blogdown. You can install it with:

## Check if you have version 0.5.7 or newer
## I actually used version 0.5.9 for this blog post
packageVersion('blogdown')

## If not, then get it!
##### If necessary:
## install.packages('devtools')
devtools::install_github('rstudio/blogdown')

You also need an up to date version of RStudio and I recommend also using R 3.4.x (or newer if you are reading this in the future). Re-start RStudio so it loads the new version of blogdown.

Step 2: open the Insert Image addin

Second, go to the Addins menu in the top section of the RStudio window and select the Insert Image blogdown addin.

Step 3: choose figure and inputs

So far the Insert Image addin looks like this:

So lets go head and select an image we want to upload. In my case, I chose an image that already exists.

You can rename the figure if you want, and if it doesn’t exist, the overwrite option goes away.

Step 4: hit done!

Lets go ahead and click done! Our text window in RStudio will insert the appropriate code for linking the image. In this case, it’s the following code:

<img src="/post/2018-03-07-blogdown-insert-image-addin_files/screenshot.png" alt="final image" width="400"/>

and this is the image:

Optional step 5

Now use the Serve Site addin and check if you like your images. You might want to change the height/widths or alternate text. You could also wrap the HTML/Markdown code around it for linking to a website.

You can also delete your original images, if for example, they are cluttering your ~/Desktop.

Conclusions

I hope that you will find this new addin as useful as I’m finding it, or even more. Plus hopefully this blog post gives you an idea of the difficulties before this addin existed. Also, I want to thank Yihui Xie for guiding me, I’ve learnt quite a bit recently. Though I will still use <- assignment operator for my own code hehe.

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, Morgan, and Huber, 2020)
blogdown (Xie, Hill, and Thomas, 2017)
devtools (Wickham, Hester, and Chang, 2019)
knitcitations (Boettiger, 2019)

as well as Yihui Xie’s help and encouragement in the form of a signed sticker and the Great Hacker title ^^. He also wrote a blog post about the Insert Image addin.

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.10. 2019. URL: https://CRAN.R-project.org/package=knitcitations.

[2] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.14.4. 2020. URL: https://github.com/Bioconductor/BiocStyle.

[3] H. Wickham, J. Hester, and W. Chang. devtools: Tools to Make Developing R Packages Easier. R package version 2.2.1. 2019. URL: https://CRAN.R-project.org/package=devtools.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.2      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/New_York            
##  date     2020-02-12                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.6.0)                    
##  backports       1.1.5   2019-10-02 [1] CRAN (R 3.6.0)                    
##  bibtex          0.4.2.2 2020-01-02 [1] CRAN (R 3.6.0)                    
##  BiocManager     1.30.10 2019-11-16 [1] CRAN (R 3.6.1)                    
##  BiocStyle     * 2.14.4  2020-01-09 [1] Bioconductor                      
##  blogdown        0.17    2019-11-13 [1] CRAN (R 3.6.1)                    
##  bookdown        0.17    2020-01-11 [1] CRAN (R 3.6.0)                    
##  callr           3.4.1   2020-01-24 [1] CRAN (R 3.6.2)                    
##  cli             2.0.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  colorout      * 1.2-1   2019-05-07 [1] Github (jalvesaq/colorout@7ea9440)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.6.0)                    
##  desc            1.2.0   2018-05-01 [1] CRAN (R 3.6.0)                    
##  devtools      * 2.2.1   2019-09-24 [1] CRAN (R 3.6.1)                    
##  digest          0.6.23  2019-11-23 [1] CRAN (R 3.6.0)                    
##  ellipsis        0.3.0   2019-09-20 [1] CRAN (R 3.6.0)                    
##  evaluate        0.14    2019-05-28 [1] CRAN (R 3.6.0)                    
##  fansi           0.4.1   2020-01-08 [1] CRAN (R 3.6.0)                    
##  fs              1.3.1   2019-05-06 [1] CRAN (R 3.6.0)                    
##  glue            1.3.1   2019-03-12 [1] CRAN (R 3.6.0)                    
##  htmltools       0.4.0   2019-10-04 [1] CRAN (R 3.6.0)                    
##  httr            1.4.1   2019-08-05 [1] CRAN (R 3.6.0)                    
##  jsonlite        1.6     2018-12-07 [1] CRAN (R 3.6.0)                    
##  knitcitations * 1.0.10  2019-09-15 [1] CRAN (R 3.6.0)                    
##  knitr           1.27    2020-01-16 [1] CRAN (R 3.6.0)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.6.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.6.0)                    
##  memoise         1.1.0   2017-04-21 [1] CRAN (R 3.6.0)                    
##  pkgbuild        1.0.6   2019-10-09 [1] CRAN (R 3.6.0)                    
##  pkgload         1.0.2   2018-10-29 [1] CRAN (R 3.6.0)                    
##  plyr            1.8.5   2019-12-10 [1] CRAN (R 3.6.0)                    
##  prettyunits     1.1.1   2020-01-24 [1] CRAN (R 3.6.2)                    
##  processx        3.4.1   2019-07-18 [1] CRAN (R 3.6.0)                    
##  ps              1.3.0   2018-12-21 [1] CRAN (R 3.6.0)                    
##  R6              2.4.1   2019-11-12 [1] CRAN (R 3.6.1)                    
##  Rcpp            1.0.3   2019-11-08 [1] CRAN (R 3.6.0)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.6.0)                    
##  remotes         2.1.0   2019-06-24 [1] CRAN (R 3.6.0)                    
##  rlang           0.4.3   2020-01-24 [1] CRAN (R 3.6.2)                    
##  rmarkdown       2.1     2020-01-20 [1] CRAN (R 3.6.0)                    
##  rprojroot       1.3-2   2018-01-03 [1] CRAN (R 3.6.0)                    
##  sessioninfo     1.1.1   2018-11-05 [1] CRAN (R 3.6.0)                    
##  stringi         1.4.5   2020-01-11 [1] CRAN (R 3.6.0)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.6.0)                    
##  testthat        2.3.1   2019-12-01 [1] CRAN (R 3.6.0)                    
##  usethis       * 1.5.1   2019-07-04 [1] CRAN (R 3.6.0)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.6.0)                    
##  xfun            0.12    2020-01-13 [1] CRAN (R 3.6.0)                    
##  xml2            1.2.2   2019-08-09 [1] CRAN (R 3.6.0)                    
##  yaml            2.2.0   2018-07-25 [1] CRAN (R 3.6.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

R markdown blog template

Sat, 17 Feb 2018 00:00:00 +0000

This blog post is mostly for myself but maybe it’s useful to others. It contains my current R markdown blog template. I initially posted this as a question at StackOverflow. Then I read how much a burden we put in Yihui Xie and decided that my current setup (copy-pasting) works just fine. In any case using blogdown with the RStudio IDE is much simpler than what I used to do in the past with jekyll or with even my prior setup with blogdown.

Bibliography setup

First I define the citation information I’ll need. By the way, I used FAQ 7 for showing the R code chunk.

    ```{r bibsetup, echo=FALSE, message=FALSE, warning=FALSE}
    ## Load knitcitations with a clean bibliography
    library('knitcitations')
    cleanbib()
    cite_options(hyperlink = 'to.doc', citation_format = 'text', style = 'html')
    
    bib <- c('knitcitations' = citation('knitcitations'),
             'blogdown' = citation('blogdown')[2])
    ```

Post content

This is where I typically start to edit since the bibliography chunk is hidden.

R image

    ```{r 'plot'}
    ## This will create the /post/*_files/ directory
    ## where you can later copy the non-R images you want to use
    ## in the blog post
    plot(1:10, 10:1)
    ```

Note that I modified the YAML portion of the post to set the figure width and height. You can also include a table of contents if you want, though it affects the summary of the post. Check the output format section of the blogdown book for more details (Xie, Hill, and Thomas, 2017), including differences between .Rmd and .Rmarkdown files. Note that you can also use a _output.yml file as described there.

output:
  blogdown::html_page:
    toc: no
    fig_width: 5
    fig_height: 5

Custom image syntax

Here I remind myself of different ways I can include external images. Check blogdown issue 239 for some background information.

Markdown syntax for custom image:

![](/post/2018-02-17-r-markdown-blog-template_files/LIBD.jpg)

HTML syntax for centering image, including a link, and re-sizing the image to a fix width of 200 px.

<center>
<a href="http://lcolladotor.github.io/"><img alt = 'some website' width='200' src='/post/2018-02-17-r-markdown-blog-template_files/LIBD.jpg' /></a>
</center>

Reproducibility

## Reproducibility info
library('devtools')
options(width = 120)
session_info()

## Session info ----------------------------------------------------------------------------------------------------------

##  setting  value                       
##  version  R version 3.4.3 (2017-11-30)
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  tz       America/New_York            
##  date     2018-03-10

## Packages --------------------------------------------------------------------------------------------------------------

##  package       * version date       source                               
##  backports       1.1.2   2017-12-13 CRAN (R 3.4.3)                       
##  base          * 3.4.3   2017-12-07 local                                
##  bibtex          0.4.2   2017-06-30 CRAN (R 3.4.1)                       
##  blogdown        0.5.10  2018-03-10 Github (lcolladotor/blogdown@471b086)
##  bookdown        0.7     2018-02-18 cran (@0.7)                          
##  colorout      * 1.2-0   2018-02-19 Github (jalvesaq/colorout@2f01173)   
##  compiler        3.4.3   2017-12-07 local                                
##  datasets      * 3.4.3   2017-12-07 local                                
##  devtools      * 1.13.5  2018-02-18 CRAN (R 3.4.3)                       
##  digest          0.6.15  2018-01-28 CRAN (R 3.4.3)                       
##  evaluate        0.10.1  2017-06-24 CRAN (R 3.4.1)                       
##  graphics      * 3.4.3   2017-12-07 local                                
##  grDevices     * 3.4.3   2017-12-07 local                                
##  htmltools       0.3.6   2017-04-28 CRAN (R 3.4.0)                       
##  httr            1.3.1   2017-08-20 CRAN (R 3.4.1)                       
##  jsonlite        1.5     2017-06-01 CRAN (R 3.4.0)                       
##  knitcitations * 1.0.8   2017-07-04 CRAN (R 3.4.1)                       
##  knitr           1.20    2018-02-20 cran (@1.20)                         
##  lubridate       1.7.3   2018-02-27 CRAN (R 3.4.3)                       
##  magrittr        1.5     2014-11-22 CRAN (R 3.4.0)                       
##  memoise         1.1.0   2017-04-21 CRAN (R 3.4.0)                       
##  methods       * 3.4.3   2017-12-07 local                                
##  plyr            1.8.4   2016-06-08 CRAN (R 3.4.0)                       
##  R6              2.2.2   2017-06-17 CRAN (R 3.4.0)                       
##  Rcpp            0.12.15 2018-01-20 CRAN (R 3.4.3)                       
##  RefManageR      0.14.20 2017-08-17 CRAN (R 3.4.1)                       
##  rmarkdown       1.9     2018-03-01 cran (@1.9)                          
##  rprojroot       1.3-2   2018-01-03 CRAN (R 3.4.3)                       
##  stats         * 3.4.3   2017-12-07 local                                
##  stringi         1.1.6   2017-11-17 CRAN (R 3.4.2)                       
##  stringr         1.3.0   2018-02-19 cran (@1.3.0)                        
##  tools           3.4.3   2017-12-07 local                                
##  utils         * 3.4.3   2017-12-07 local                                
##  withr           2.1.1   2017-12-19 CRAN (R 3.4.3)                       
##  xfun            0.1     2018-01-22 CRAN (R 3.4.3)                       
##  xml2            1.2.0   2018-01-24 CRAN (R 3.4.3)                       
##  yaml            2.1.18  2018-03-08 cran (@2.1.18)

References

Citations made with knitcitations (Boettiger, 2017) and blog built using blogdown (Xie, Hill, and Thomas, 2017).

## Chunk normaly with options:  results = 'asis', echo = FALSE, cache = FALSE
## Print bibliography
bibliography(style = 'html')

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.8. 2017. URL: https://CRAN.R-project.org/package=knitcitations.

[2] Y. Xie, A. P. Hill and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Updating R

Thu, 04 May 2017 00:00:00 +0000

As you might know by now, the latest R version was recently released (R 3.4.0). That means that you are highly encouraged to update your R installation. There are several ways to do this some of which are documented in these other blog posts: Tal Galili, 2013, Kris Eberwein, 2015. You would think that it’s just a matter of downloading the latest R installer for your OS, installing it, and continuing your analysis. The reality is a bit more complicated. The following short steps will make your life easier.

Save your list of currently installed packages

If you want to continue using R with all the packages you previously had installed, the best way is to save the list of packages you currently have before installing a new R version. You can do so with these lines of code:

## Change accordingly
list_dir <- '/Users/lcollado/Dropbox/Computing/R'

## Get the list of installed packages
installed <- dir(.libPaths())

## Save the list for later use
save(installed, file = file.path(list_dir, paste0(Sys.Date(), '-installed.Rdata')))

## Explore the list
head(installed)

## [1] "abind"   "acepack" "ada"     "AER"     "affy"    "affyio"

length(installed)

## [1] 611

Install latest R

Ok, now you have a list of installed packages. It’s like a restore point. Next, you need to get the latest R installer for your OS from CRAN and install the latest R. For example, for a Mac that would be R-3.4.0.pkg. Install it as usual.

Restore your packages

By now you have a new R version installed but without all your favorite packages. So, how do you install them? You just need to open your latest list of installed packages and install them. Now, I’m a Bioconductor user which means that some of my packages are not on CRAN. But the following code will work for you even if all the packages you use are from CRAN.

## Change accordingly
list_dir <- '/Users/lcollado/Dropbox/Computing/R'

## Find the corresponding Rdata files
previous <- dir(path = list_dir, pattern = 'installed.Rdata')

## Load the latest one
load(file.path(list_dir, previous[length(previous)]))

## Just checking it
head(installed)

## [1] "abind"   "acepack" "ada"     "AER"     "affy"    "affyio"

Next, get the list of current R packages you have installed. Every new R installation comes with a few of them (the base packages). You don’t need to install those.

current <- dir(.libPaths())

Finally, install the missing packages

## For Bioconductor and CRAN packages
install.packages("BiocManager")
BiocManager::install(installed[!installed %in% current])

and now you can continue on with your analysis 😄 You didn’t even need to figure out the best order to install the packages!

GitHub packages

Some of your favorite R packages might only exist via GitHub. This list is likely short since most packages get distributed via CRAN. But if that’s the case, you can see which packages are missing by running:

## Check which packages are missing
current_post_installation <- dir(.libPaths())
installed[!installed %in% current_post_installation]

For example, in my case I use the colorout package which lives only in GitHub. I have to install that one manually:

install.packages('devtools')
library('devtools')
install_github("jalvesaq/colorout")

Other times a package might not be compiling for the new R version or might no longer be supported (defunct).

Misc for Bioconductor developers

If you are a Bioconductor developer or are planning on becoming one, then you need 2 versions of R at all times. One R for the bioc-release branch and another one for the bioc-devel branch. Sometimes it’s the same R version sometimes it’s not depending on the month of the year. Right now, Bioc-release (3.5) uses R 3.4.0 and Bioc-devel (3.6) also uses R 3.4.0. R Switch for Mac users will be your friend. I can’t find the old bioc-devel mailing list thread where I first learned this, but the idea is to download the latest R tar ball, change the name from 3.4 to something else (3.4devel in my case), put it back together into a tar ball and then use this tar ball to install a second R version.

## Download latest R tarball
wget http://r.research.att.com/el-capitan/R-3.4-branch/R-3.4-branch-el-capitan-sa-x86_64.tar.gz

## Un-tar it
tar -xvf R-3.4-branch-el-capitan-sa-x86_64.tar.gz

## Renamed files from 3.4 to 3.4 devel
mv Library/Frameworks/R.framework/Versions/3.4 Library/Frameworks/R.framework/Versions/3.4devel

## Put it back in a tar ball
tar -cvzf Rlib.tgz Library

## Install it
sudo tar fvxz Rlib.tgz -C /

There you go:

Reproducibility

## Reproducibility info
library('devtools')
options(width = 120)
session_info()

## Session info -----------------------------------------------------------------------------------------------------------

##  setting  value                       
##  version  R version 3.4.0 (2017-04-21)
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  tz       America/New_York            
##  date     2017-05-04

## Packages ---------------------------------------------------------------------------------------------------------------

##  package       * version date       source        
##  BiocInstaller * 1.27.2  2017-05-04 Bioconductor  
##  devtools      * 1.12.0  2016-12-05 CRAN (R 3.4.0)
##  digest          0.6.12  2017-01-27 CRAN (R 3.4.0)
##  evaluate        0.10    2016-10-11 cran (@0.10)  
##  knitr         * 1.15.1  2016-11-22 cran (@1.15.1)
##  magrittr        1.5     2014-11-22 cran (@1.5)   
##  memoise         1.1.0   2017-04-21 CRAN (R 3.4.0)
##  stringi         1.1.5   2017-04-07 cran (@1.1.5) 
##  stringr         1.2.0   2017-02-18 cran (@1.2.0) 
##  withr           1.0.2   2016-06-20 CRAN (R 3.4.0)

Want more?

Check other @jhubiostat student and alumni blogs at Bmore Biostats as well as topics on #rstats.

How to ask for help for Bioconductor packages

Mon, 06 Mar 2017 00:00:00 +0000

tl;dr Please post your question at the Bioconductor support website https://support.bioconductor.org/ and check the posting guide http://www.bioconductor.org/help/support/posting-guide/. It’s important that you provide reproducible code and information about your R session.

Recently I have been getting more questions about several packages I maintain. It’s great to see more interest from users, but at the same time most questions lack the information I need to help the users. I have also gotten most of the questions via email, which is why I am writing this post. As of today, I will no longer answer questions related to my Bioconductor packages via personal emails. This might sound harsh, but hopefully the rest of this post will convince you that it’s the best thing to do. You might also be interested in the basics of using derfinder, regionReport or recount, among others.

The Bioconductor project is a community project and it benefits from users interacting in public venues. When a user asks a question at the Bioconductor support website, they are providing information that future users might be interested in. That is, the user (U1) is contributing information to the overall documentation around the Bioconductor package they are asking a question about. Ideally, a new user (U2) can then read through the question U1 wrote, check the solution, and move on. This is one of the main reasons why we (developers) want questions to be well documented. There are a couple of quick things that U1 can check that will make their question much more useful to the community.

Session information

One of the strengths of Bioconductor is that all the packages have vignettes and lots of documentation. The packages are also checked regularly and must pass some tests. That also means that packages can change frequently, at least more frequently than CRAN packages. There’s also the added complexity that at any given point in time there is a release branch and a development branch. This means that there are many variables and saying that you are using the “latest version” doesn’t mean much to the developer. All of this information and more is part of the R session information. That is why I and others request users to post their session information. It’s very easy to get, simply run the following code:

## Install devtools if needed
# install.packages('devtools')

## Reproducibility info
library('devtools')
options(width = 120)
session_info()

The output might be too long to post in the Bioconductor support website. The easy solution is to save the information you want displayed in a gist. Then simply add the gist link in your question. Note that you need to have the link under “text” formatting and not “code”.

Code to reproduce the error

If U1 includes the session information, their question will be pretty good, but not ideal yet. Many of the questions I’ve been asked do not include code for me to figure out the exact steps of what they were doing. A lot of times I can infer pieces of what they were doing from their description of the problem. But doing so takes quite a bit of my time and effort, and is still not perfect. Now imagine that U2 is reading through the question: they would probably get lost!

There is a wide range of things that U1 could have done. To help the developer, the best thing is for the user to include the code that lead to the error. The code should include how the data was loaded, so that the developer can run it themselves and check in more detail what went wrong. This means providing a small subset of the data or using some publicly available data.

I realize that writing code that reproduces the error is not easy. But it helps a lot for learning more about R and Bioconductor. I can tell you that I went through the same process, and in my experience you can find out what you are doing wrong by writing the reproducible code.

Extra

Here are some other tips that are useful.

If you run traceback() immediately after getting the error and include the output in your question, that would be great. It makes it easier to check at what point the code failed and produced the error.
Recently when I ask questions myself, I include the “non-evaluated code” (clean code in your script) and “evaluated code” (think of the R console: a mix of code and output). The non-evaluated code makes it easier for others to copy-paste the code into their R session without having to deal with any formatting issues ( example).
If you encounter a new error, post a new question instead of “replying” to the first one.
Introduce yourself.
Be polite.

By now you should be ready to post some great questions! Thanks for contributing to the Bioconductor community.

Want more?

Check other @jhubiostat student and alumni blogs at Bmore Biostats as well as topics on #rstats.

Easily explore a table with shinycsv

Fri, 20 Jan 2017 00:00:00 +0000

Have you ever had to explore a table with data? I believe the answer is yes for most people that work at a computer or even just use it for communicating with their friends and family. Tables of data pop up everywhere, for example in personal finance. Websites like Mint.com allow you to download your transactions in a CSV file called transactions.csv. CSV is one of the many formats for storing tables and most likely when you try to open the transactions.csv file, it will open with Excel. Now, can you make a quick figure of one of your columns in your table?

Some will answer yes, others no. The basic issue is that it’s not super easy to explore your data in Excel or similar programs. Wait, shouldn’t it be easy? 😕

What if you want to subset your data and want to re-make the plot? How about getting some simple statistics like the mean or frequency of some categories for a given variable? 😨 These are some of the immediate tasks that are helpful when exploring data. Visually, making figures with two variables is also very common.

Programmers and experts in Excel, Stata, R among other options can perform these data explorations. It might take them a little bit of time to write the code or remember it or use the user interface menu of their program of choice. But what about everyone else?

At the Lieber Institute for Brain Development where I work, it’s common for us to exchange data in tables, and thus explore data. That’s why we created shinycsv (Collado-Torres, Semick, and Jaffe, 2016). It’s an R package (R Core Team, 2016) that contains a shiny (Chang, Cheng, Allaire, Xie, et al., 2017) application that allows users to interactively explore a table.

Installing R is a pretty high bar, that’s why we are hosting this application at https://jhubiostatistics.shinyapps.io/shinycsv/. Try it out!

shinycsv application

The application includes data about cars to demonstrate what it can do. It’s a small data set that is commonly used for demonstration purposes. Anyhow, in the application you’ll notice a few tabs.

The application shows the raw data in an interactive table that allows you to subset the observations by some criteria, search in the table, and sort in different ways. The raw summary tab shows quick statistical summaries which depend on the variable type (numerical, categorical, etc). If you interacted with the table in raw data then the summaries at raw summary will be based on the subset you selected.

The one variable and two variables tabs are for making figures based on one or two variables at a time. The code in shinycsv tries to guess what’s the best figure for a given type of variable and in case that you are interested in learning R, it also shows the exact code you can use to reproduce the figure in your computer. We added this feature to excite users about learning R. And it’s useful for advanced users too that might want to customize the resulting figures. Hm…, you don’t like the colors we chose for the figure? Well go to plot colors, choose another color, and come back to see your new figure with the color of your choosing. 😄

Hm… but what if you don’t have a CSV file? Well, shinycsv can handle many different tables thanks to rio (Chan, Chan, Leeper, and Becker, 2016). Even Excel sheets! 😉

So, go ahead and test it out! We’ll be glad to hear your feedback at LieberInstitute/shinycsv.

Notes

Note that when I referred to tables earlier, I referred to square tables with different variables (age, height, weight, etc) as columns as observations as rows. That is, Excel files with a single sheet with no comments or figures inside the Excel file.
Are you interested in learning more about R and shiny? Maybe you’ll want to take a look at the showcase mode version of the application.
If you use shinycsv::explore() locally, the file size limit is increased to 500 MB. Although at that point you might want to consider using R or another programming language.
What about casting variables? If you want to have fine control about casting the variables, save your data in a RData file. Sure, this requires an R user.

Reproducibility

## Reproducibility info
library('devtools')
options(width = 120)
session_info()

## Session info -----------------------------------------------------------------------------------------------------------

##  setting  value                                             
##  version  R Under development (unstable) (2016-10-26 r71594)
##  system   x86_64, darwin13.4.0                              
##  ui       X11                                               
##  language (EN)                                              
##  collate  en_US.UTF-8                                       
##  tz       America/New_York                                  
##  date     2017-01-20

## Packages ---------------------------------------------------------------------------------------------------------------

##  package       * version  date       source        
##  bibtex          0.4.0    2014-12-31 CRAN (R 3.4.0)
##  bitops          1.0-6    2013-08-17 CRAN (R 3.4.0)
##  devtools      * 1.12.0   2016-12-05 CRAN (R 3.4.0)
##  digest          0.6.11   2017-01-03 CRAN (R 3.4.0)
##  evaluate        0.10     2016-10-11 CRAN (R 3.4.0)
##  httr            1.2.1    2016-07-03 CRAN (R 3.4.0)
##  knitcitations * 1.0.7    2015-10-28 CRAN (R 3.4.0)
##  knitr         * 1.15.1   2016-11-22 CRAN (R 3.4.0)
##  lubridate       1.6.0    2016-09-13 CRAN (R 3.4.0)
##  magrittr        1.5      2014-11-22 CRAN (R 3.4.0)
##  memoise         1.0.0    2016-01-29 CRAN (R 3.4.0)
##  plyr            1.8.4    2016-06-08 CRAN (R 3.4.0)
##  R6              2.2.0    2016-10-05 CRAN (R 3.4.0)
##  Rcpp            0.12.9   2017-01-14 CRAN (R 3.4.0)
##  RCurl           1.95-4.8 2016-03-01 CRAN (R 3.4.0)
##  RefManageR      0.13.1   2016-11-13 CRAN (R 3.4.0)
##  RJSONIO         1.3-0    2014-07-28 CRAN (R 3.4.0)
##  stringi         1.1.2    2016-10-01 CRAN (R 3.4.0)
##  stringr         1.1.0    2016-08-19 CRAN (R 3.4.0)
##  withr           1.0.2    2016-06-20 CRAN (R 3.4.0)
##  XML             3.98-1.5 2016-11-10 CRAN (R 3.4.0)

References

Citations made with knitcitations (Boettiger, 2015).

[1] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.7. 2015. URL: https://CRAN.R-project.org/package=knitcitations.

[2] C. Chan, G. C. Chan, T. J. Leeper and J. Becker. rio: A Swiss-army knife for data file I/O. R package version 0.4.16. 2016.

[3] W. Chang, J. Cheng, J. Allaire, Y. Xie, et al. shiny: Web Application Framework for R. R package version 1.0.0. 2017. URL: https://CRAN.R-project.org/package=shiny.

[4] L. Collado-Torres, S. Semick and A. E. Jaffe. shinycsv: Explore a table interactively in a shiny application. R package version 0.99.7. 2016. URL: https://github.com/LieberInstitute/shinycsv.

[5] R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. Vienna, Austria, 2016. URL: https://www.R-project.org/.

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

Finding possible class schedules

Tue, 02 Aug 2016 00:00:00 +0000

Over the weekend my brother wanted to figure out his class schedule for the next semester. He is a veterinary medicine and zootechnology student at UNAM. For this upcoming semester there is a set of classes he has to take and each has 8 or so instructor options. The website where he finds the class times lists about 8 pre-constructed class schedules. So he normally finds one he likes quite a bit, and then manually starts checking if he can change X instructor for Y for a given class. He does this based on the referalls and information he has gathered about the instructors, plus he factors in whether it’d be an overall better schedule. For example, he might prefer to have a packed Tuesday if that means that he can leave early on Friday and avoid classes on Saturday.

The problem is that it’s very easy to make a mistake. You (well he) gets all excited thinking that he’s found the perfect schedule. Only to then realize that there is a conflict between two classes. Or that the practical portion of a class is in a location one hour away from the university, meaning that the schedule he has selected won’t work. This process is very frustrating.

I was watching him and I started to think if I could help him with some code. Turns out that it was straightforward to write some code to find which options are valid. Once I wrote a test case, it took us like half an hour to fill out the data. I know that tomorrow is when he and his classmates start registering for classes, so this information might help his classmates.

First, I define some helper functions. These are rather straightforward but I’ll be using them later on. For example, dias() is just there for typing less.

## Helper functions
dias <- function(d, i) {
    paste0(d, i)
}

extract <- function(m, p) {
    m[[p]]
}

extract_names <- function(m, p) {
    names(m)[p]
}

Next comes the input information. I organized it in a set of nested list objects. The schedule is stored as a character vector. For example, Lucia Eliana’s class meets on Wednesdays (__M__iercoles in Spanish) from 9 to 11 am. I only keep the starting hours (9 and 10 am) because otherwise the code won’t detect valid opitons that include another class that starts at 11 am. For classes that are 1 hour away from the university, we included 1 hour before and 1 hour after the class.

## Input class/prof info and schedule
materias <- list(
    repro = list(
        'lucia eliana' = c(dias('m', 9:10), dias('j', 9:10), dias('v', 8:13)),
        'esquivel lacroix' = c(dias('l', 14:15), dias('ma', 12:18), dias('m', 14:15)),
        'ismael porras' = c(dias('l', 9:10), dias('m', 8:14), dias('j', 9:10)),
        'esquivel lacroix 2' = c(dias('l', 14:15), dias('m', 14:15), dias('v', 12:18)),
        'salvador galina' = c(dias('ma', 8:13), dias('j', 9:10), dias('v', 9:10)),
        'alberto balcazar' = c(dias('l', 15:16), dias('j', 12:18), dias('v', 14:15)),
        'ana myriam boeta' = c(dias('l', 8:13), dias('m', 10:11), dias('v', 9:10)),
        'rafael eduardo paz' = c(dias('l', 11:17), dias('j', 14:15), dias('v', 16:17)),
        'juan heberth' = c(dias('ma', 9:10), dias('m', 11:17), dias('v', 11:12)),
        'vicente octavio mejia' = c(dias('ma', 8:17))
    ),
    economia = list(
        'valentin efren espinoza' = c(dias('l', 8:10), dias('ma', 9:11)),
        'maria del pilar velazquez' = c(dias('l', 16:18), dias('m', 16:18)),
        'arturo alonso pesado' = c(dias('l', 11:13), dias('j', 11:13)),
        'laura mendez' = c(dias('ma', 13:15), dias('m', 18:20)),
        'laura mendez 2' = c(dias('j', 11:13), dias('v', 11:13)),
        'manuela garcia' = c(dias('l', 17:19), dias('v', 16:18)),
        'francisco alejandro' = c(dias('ma', 7:9), dias('j', 7:9)),
        'isaac reyes' = c(dias('m', 13:15), dias('v', 13:15)),
        'jose luis tinoco' = c(dias('ma', 12:14), dias('m', 9:11)),
        'isaac reyes 2' = c(dias('l', 14:16), dias('ma', 14:16))
        
    ),
    bacterianas = list(
        'jose luis gutierrez' = dias('s', 8:11),
        'rodrigo mena' = c(dias('ma', 18:19), dias('j', 18:19)),
        'beatriz arellano' = c(dias('l', 7:8), dias('ma', 10:11)),
        'de la pena, ramirez ortega' = c(dias('j', 18:19), dias('v', 18:19)),
        'ramirez ortega' = c(dias('m', 7:8), dias('j', 7:8)),
        'rodrigo mena 2' = c(dias('ma', 16:17), dias('m', 16:17)),
        'de la pena' = dias('s', 8:11),
        'efren diaz aparicio' = dias('s', 8:11),
        'lucia del carmen favila' = dias('s', 8:11)
    ),
    parasitarias = list(
        'cintli martinez' = c(dias('j', 16:17), dias('v', 18:20)),
        'osvaldo froylan' = c(dias('ma', 18:19), dias('j', 18:20)),
        'maria quintero, agustin perez' = c(dias('ma', 13:14), dias('m', 7:9)),
        'maria quintero' = c(dias('m', 16:18), dias('j', 16:17)),
        'evangelina romero' = c(dias('ma', 7:8), dias('v', 7:9)),
        'guadarrama 01' = c(dias('m', 7:8), dias('j', 11:13)),
        'guadarrama 03' = c(dias('ma', 13:15), dias('v', 7:8)),
        'guadarrama 04' = c(dias('l', 16:17), dias('ma', 18:20)),
        'guadarrama 05' = c(dias('l', 7:9), dias('j', 7:8))
    ),
    diagnosticas = list(
        '1701' = c(dias('l', 11:13), dias('m', 11:16)),
        '1702' = c(dias('j', 13:15), dias('v', 13:18)),
        '1703' = c(dias('ma', 7:9), dias('v', 8:13)),
        '1704' = c(dias('l', 18:20), dias('j', 13:18)),
        '1705' = c(dias('l', 11:13), dias('m', 7:11)),
        '1706' = c(dias('ma', 15:17), dias('m', 15:19)),
        '1707' = c(dias('ma', 10:12), dias('j', 10:15)),
        '1708' = c(dias('l', 18:20), dias('ma', 15:19)),
        '1709' = c(dias('l', 11:13), dias('j', 8:13)),
        '1711' = c(dias('j', 13:15), dias('v', 10:13))
    )
)

Now that the input information is complete, I use expand.grid() to find out all the different possible options.

## Get all the options
options <- expand.grid(lapply(materias, function(x) { seq_len(length(x))}))
dim(options)

## [1] 81000     5

There’s 81,00 of them including the classes that meet on Saturday. You can see why it’s a frustrating process to find which combination of classes work when doing this manually.

Next, I explore all these options to find those that are valid, meaning that none of the classes overlap. I do this by finding which options have no duplicated hours from the character vectors defined earlier. Nothing fancy.

valid <- apply(options, 1, function(input) {
    info <- mapply(extract, materias, input)
    !any(duplicated(unlist(info)))
})

Now that I have the valid options, I can find the names of the instructors for them. There’s 2,847 valid schedules in the end, out of the 81,000. That’s 3.5 percent!

valid_prof <- apply(options[valid, ], 1, function(input) {
    mapply(extract_names, materias, input)
})
ncol(valid_prof)

## [1] 2847

You can search the interactive version here to select only the options with a given instructor. For example, in my brother’s case there are 30 valid options once he decided to prioritize two instructors as shown in the non-interactive table below.

## Ideally, this code would create an interactive table, but it doesn't work for some reason:
#library('DT')
#datatable(t(valid_prof), options = list(pagingType='full_numbers', pageLength=10), rownames = FALSE)
valid_prof <- t(valid_prof)
rownames(valid_prof) <- seq_len(nrow(valid_prof))
top_options <- valid_prof[valid_prof[, 1] == 'lucia eliana' & valid_prof[, 2] %in% c('isaac reyes', 'isaac reyes 2'), ]
kable(top_options, format = 'markdown', row.names = TRUE)

	repro	economia	bacterianas	parasitarias	diagnosticas
5	lucia eliana	isaac reyes 2	jose luis gutierrez	cintli martinez	1701
11	lucia eliana	isaac reyes 2	rodrigo mena	cintli martinez	1701
14	lucia eliana	isaac reyes 2	beatriz arellano	cintli martinez	1701
19	lucia eliana	isaac reyes 2	ramirez ortega	cintli martinez	1701
25	lucia eliana	isaac reyes 2	de la pena	cintli martinez	1701
31	lucia eliana	isaac reyes 2	efren diaz aparicio	cintli martinez	1701
37	lucia eliana	isaac reyes 2	lucia del carmen favila	cintli martinez	1701
46	lucia eliana	isaac reyes 2	jose luis gutierrez	osvaldo froylan	1701
50	lucia eliana	isaac reyes 2	beatriz arellano	osvaldo froylan	1701
58	lucia eliana	isaac reyes 2	ramirez ortega	osvaldo froylan	1701
67	lucia eliana	isaac reyes 2	de la pena	osvaldo froylan	1701
76	lucia eliana	isaac reyes 2	efren diaz aparicio	osvaldo froylan	1701
85	lucia eliana	isaac reyes 2	lucia del carmen favila	osvaldo froylan	1701
123	lucia eliana	isaac reyes 2	jose luis gutierrez	guadarrama 01	1701
130	lucia eliana	isaac reyes 2	rodrigo mena	guadarrama 01	1701
134	lucia eliana	isaac reyes 2	beatriz arellano	guadarrama 01	1701
137	lucia eliana	isaac reyes 2	de la pena, ramirez ortega	guadarrama 01	1701
144	lucia eliana	isaac reyes 2	de la pena	guadarrama 01	1701
151	lucia eliana	isaac reyes 2	efren diaz aparicio	guadarrama 01	1701
158	lucia eliana	isaac reyes 2	lucia del carmen favila	guadarrama 01	1701
209	lucia eliana	isaac reyes 2	jose luis gutierrez	guadarrama 05	1701
217	lucia eliana	isaac reyes 2	rodrigo mena	guadarrama 05	1701
222	lucia eliana	isaac reyes 2	de la pena, ramirez ortega	guadarrama 05	1701
232	lucia eliana	isaac reyes 2	de la pena	guadarrama 05	1701
242	lucia eliana	isaac reyes 2	efren diaz aparicio	guadarrama 05	1701
252	lucia eliana	isaac reyes 2	lucia del carmen favila	guadarrama 05	1701
872	lucia eliana	isaac reyes 2	jose luis gutierrez	guadarrama 05	1704
885	lucia eliana	isaac reyes 2	de la pena	guadarrama 05	1704
894	lucia eliana	isaac reyes 2	efren diaz aparicio	guadarrama 05	1704
903	lucia eliana	isaac reyes 2	lucia del carmen favila	guadarrama 05	1704

Reproducibility

## Reproducibility info
library('devtools')
session_info()

## Session info --------------------------------------------------------------

##  setting  value                       
##  version  R version 3.3.0 (2016-05-03)
##  system   x86_64, mingw32             
##  ui       RStudio (0.99.902)          
##  language (EN)                        
##  collate  English_United States.1252  
##  tz       America/Mexico_City         
##  date     2016-08-02

## Packages ------------------------------------------------------------------

##  package   * version date       source        
##  devtools  * 1.12.0  2016-06-24 CRAN (R 3.3.1)
##  digest      0.6.9   2016-01-08 CRAN (R 3.3.0)
##  evaluate    0.9     2016-04-29 CRAN (R 3.3.0)
##  formatR     1.4     2016-05-09 CRAN (R 3.3.0)
##  highr       0.6     2016-05-09 CRAN (R 3.3.0)
##  knitr     * 1.13    2016-05-09 CRAN (R 3.3.0)
##  magrittr    1.5     2014-11-22 CRAN (R 3.3.0)
##  memoise     1.0.0   2016-01-29 CRAN (R 3.3.0)
##  rsconnect   0.4.3   2016-05-02 CRAN (R 3.3.0)
##  stringi     1.1.1   2016-05-27 CRAN (R 3.3.0)
##  stringr     1.0.0   2015-04-30 CRAN (R 3.3.0)
##  withr       1.0.2   2016-06-20 CRAN (R 3.3.1)

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

Are you doing parallel computations in R? Then use BiocParallel

Mon, 07 Mar 2016 00:00:00 +0000

It’s the morning of the first day of oral conferences at #ENAR2016. I feel like I have a spidey sense since I woke up 3 min after an email from Jeff Leek; just a funny coincidence. Anyhow, I promised Valerie Obenchain at #Bioc2014 that I would write a post about one of my favorite Bioconductor packages: BiocParallel (Morgan, Obenchain, Lang, and Thompson, 2016). By now it’s on the top 5% of downloaded Bioconductor packages, so many people know about it or are unaware that their favorite package uses it behind the scenes.

While I haven’t blogged about BiocParallel yet, I did give a presentation about it at our computing club back in April 2nd, 2015. See it here ( source). I’m going to follow its structure in this post.

Parallel computing

Before even thinking about using BiocParallel you have to decide whether parallel computing is the thing you need.

While I’m not talking about cloud computing, I still find this picture funny.

There’s different types of parallel computing, but what I’m referring to here is called embarrassingly parallel where you have a task to do for a set of inputs, you split your inputs into subsets and perform the task on these subsets. Performing this task for one input a a time is called serial programming and it’s what we do in most cases when using functions like lapply() or for loops.

plot(y = 10 / (1:10), 1:10, xlab = 'Number of cores', ylab = 'Time',
    main = 'Ideal scenario', type = 'o', col = 'blue',
    cex = 2, cex.axis = 2, cex.lab = 1.5, cex.main = 2, pch = 16)

You might be running a simulation for a different set of parameters (a parameter grid) and running each simulation could take some time. Parallel computing can help you speed up this problem. In the ideal scenario, the higher number of computing cores (units that evaluate subsets of your inputs) the less time you need to run your full analysis.

plot(y = 10 / (1:10), 1:10, xlab = 'Number of cores', ylab = 'Time',
    main = 'Reality', type = 'o', col = 'blue',
    cex = 2, cex.axis = 2, cex.lab = 1.5, cex.main = 2, pch = 16)
lines(y = 10 / (1:10) * c(1, 1.05^(2:10) ), 1:10, col = 'red',
    type = 'o', cex = 2)

However, in reality parallel computing is not cost-free. It involves some communication costs, like sending the data to the cores, aggregating the results in a way that you can then easily use, among other things. So, it’ll be a bit slower than the ideal scenario but you can potentially still greatly reduce the overall time.

Having said all of the above, lets say that you now want to do some parallel computing in R. Where do you start? A pretty good place to start is the CRAN Task View: High-Performance and Parallel Computing with R. There you’ll find a lot of information about different packages that enable you to do parallel computing with R.

But you’ll soon be lost in a sea of new terms.

Why use BiocParallel?

It’s simple to use.
You can try different parallel backends without changing your code.
You can use it to submit cluster jobs.
You’ll have access to great support from the Bioconductor developer team.

Those are the big reasons of why I use BiocParallel. But let me go through them a bit more slowly.

Birthday example

I’m going to use as an example the birthday problem where you want to find out empirically the probability that two people share the same birthday in a room.

birthday <- function(n) {
    m <- 10000
    x <- numeric(m)
    for(i in seq_len(m)) {
        b <- sample(seq_len(365), n, replace = TRUE)
        x[i] <- ifelse(length(unique(b)) == n, 0, 1)
    }
    mean(x)
}

Naive birthday code

Once you have written the code for it, you can then use lapply() or a for loop to calculate the results.

system.time( lapply(seq_len(100), birthday) )

##    user  system elapsed 
##  25.610   0.442  27.430

Takes around 25 seconds.

Via doMC

If you looked at CRAN Task View: High-Performance and Parallel Computing with R you might have found the doMC (Analytics and Weston, 2015).

It allows you to run computations in parallel as shown below.

library('doMC')

## Loading required package: foreach

## Loading required package: iterators

## Loading required package: parallel

registerDoMC(2)
system.time( x <- foreach(j = seq_len(100)) %dopar% birthday(j) )

##    user  system elapsed 
##  12.819   0.246  13.309

While it’s a bit faster, the main problem is that you had to change your code in order to be able to use it.

With BiocParallel

This is how you would run things with BiocParallel.

library('BiocParallel')
system.time( y <- bplapply(seq_len(100), birthday) )

##    user  system elapsed 
##   0.021   0.011  16.095

The only change here is using bplapply() instead of lapply(), so just 2 characters. Well, that and loading the BiocParallel package.

BiocParallel’s advantages

There are many computation backends and one of the strongest features of BiocParallel is that it’s easy to switch between them. For example, my computer can run the following options:

registered()

## $MulticoreParam
## class: MulticoreParam 
##   bpjobname:BPJOB; bpworkers:2; bptasks:0; bptimeout:Inf; bpRNGseed:; bpisup:FALSE
##   bplog:FALSE; bpthreshold:INFO; bplogdir:NA
##   bpstopOnError:FALSE; bpprogressbar:FALSE
##   bpresultdir:NA
## cluster type: FORK 
## 
## $SnowParam
## class: SnowParam 
##   bpjobname:BPJOB; bpworkers:2; bptasks:0; bptimeout:Inf; bpRNGseed:; bpisup:FALSE
##   bplog:FALSE; bpthreshold:INFO; bplogdir:NA
##   bpstopOnError:FALSE; bpprogressbar:FALSE
##   bpresultdir:NA
## cluster type: SOCK 
## 
## $SerialParam
## class: SerialParam 
##   bplog:FALSE; bpthreshold:INFO
##   bpcatchErrors:FALSE

If I was doing this in our computing cluster, I would see even more options.

Now lets say that I want to test different computation backends, or even run things in serial mode so I can trace a bug down more easily. Well, all I have to do is change the BPPARAM argument as shown below.

## Test in serial mode
system.time( y.serial <- bplapply(1:10, birthday,
    BPPARAM = SerialParam()) )

##    user  system elapsed 
##   2.577   0.033   2.733

## Try Snow
system.time( y.snow <- bplapply(1:10, birthday, 
    BPPARAM = SnowParam(workers = 2)) )

##    user  system elapsed 
##   0.027   0.006   2.436

Talking about computing clusters, you might be interested in using BatchJobs (Bischl, Lang, Mersmann, Rahnenführer, et al., 2015) just like Prasad Patil did for his PhD work. Well, with BiocParallel you can also chose to use the BatchJobs backend. I have code showing this at the presentation I referenced earlier.

Where do I start?

If you are convinced about using BiocParallel, which I hope you are by now, check out the Introduction to BiocParallel vignette available at BiocParallel’s landing page. It explains in more detail how to use it and it’s rich set of features. But if you just want to jump right in and start playing around with it, install it by running the following code:

install.packages("BiocManager")
BiocManager::install("BiocParallel")

Conclusions

Like I said earlier, BiocParallel is simple to use and has definite advantages over other solutions.

You can try different parallel backends without changing your code.
You can use it to submit cluster jobs.
You’ll have access to great support from the Bioconductor developer team. See the biocparallel tag at the support website.

Have fun using it!

Reproducibility

## Reproducibility info
library('devtools')
session_info()

## Session info --------------------------------------------------------------

##  setting  value                       
##  version  R version 3.2.2 (2015-08-14)
##  system   x86_64, darwin13.4.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  tz       America/Chicago             
##  date     2016-03-07

## Packages ------------------------------------------------------------------

##  package        * version  date       source        
##  bibtex           0.4.0    2014-12-31 CRAN (R 3.2.0)
##  BiocParallel   * 1.4.3    2015-12-16 Bioconductor  
##  bitops           1.0-6    2013-08-17 CRAN (R 3.2.0)
##  codetools        0.2-14   2015-07-15 CRAN (R 3.2.2)
##  devtools       * 1.10.0   2016-01-23 CRAN (R 3.2.3)
##  digest           0.6.9    2016-01-08 CRAN (R 3.2.3)
##  doMC           * 1.3.4    2015-10-13 CRAN (R 3.2.0)
##  evaluate         0.8      2015-09-18 CRAN (R 3.2.0)
##  foreach        * 1.4.3    2015-10-13 CRAN (R 3.2.0)
##  formatR          1.2.1    2015-09-18 CRAN (R 3.2.0)
##  futile.logger    1.4.1    2015-04-20 CRAN (R 3.2.0)
##  futile.options   1.0.0    2010-04-06 CRAN (R 3.2.0)
##  httr             1.1.0    2016-01-28 CRAN (R 3.2.3)
##  iterators      * 1.0.8    2015-10-13 CRAN (R 3.2.0)
##  knitcitations  * 1.0.7    2015-10-28 CRAN (R 3.2.0)
##  knitr          * 1.12.3   2016-01-22 CRAN (R 3.2.3)
##  lambda.r         1.1.7    2015-03-20 CRAN (R 3.2.0)
##  lubridate        1.5.0    2015-12-03 CRAN (R 3.2.3)
##  magrittr         1.5      2014-11-22 CRAN (R 3.2.0)
##  memoise          1.0.0    2016-01-29 CRAN (R 3.2.3)
##  plyr             1.8.3    2015-06-12 CRAN (R 3.2.1)
##  R6               2.1.2    2016-01-26 CRAN (R 3.2.3)
##  Rcpp             0.12.3   2016-01-10 CRAN (R 3.2.3)
##  RCurl            1.95-4.7 2015-06-30 CRAN (R 3.2.1)
##  RefManageR       0.10.6   2016-02-15 CRAN (R 3.2.3)
##  RJSONIO          1.3-0    2014-07-28 CRAN (R 3.2.0)
##  snow             0.4-1    2015-10-31 CRAN (R 3.2.0)
##  stringi          1.0-1    2015-10-22 CRAN (R 3.2.0)
##  stringr          1.0.0    2015-04-30 CRAN (R 3.2.0)
##  XML              3.98-1.3 2015-06-30 CRAN (R 3.2.0)

References

Citations made with knitcitations (Boettiger, 2015).

[1] R. Analytics and S. Weston. doMC: Foreach Parallel Adaptor for 'parallel'. R package version 1.3.4. 2015. URL: http://CRAN.R-project.org/package=doMC.

[2] B. Bischl, M. Lang, O. Mersmann, J. Rahnenführer, et al. “BatchJobs and BatchExperiments: Abstraction Mechanisms for Using R in Batch Environments”. In: Journal of Statistical Software 64.11 (2015), pp. 1–25. URL: http://www.jstatsoft.org/v64/i11/.

[3] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.7. 2015. URL: http://CRAN.R-project.org/package=knitcitations.

[4] M. Morgan, V. Obenchain, M. Lang and R. Thompson. BiocParallel: Bioconductor facilities for parallel evaluation. R package version 1.4.3. 2016.

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

Teaching a short topic to beginner R users

Tue, 07 Jul 2015 00:00:00 +0000

A couple weeks ago I was given the opportunity to teach a 1 hr 30 min slot of an introduction to R course. In the past, I’ve taught lectures for similar courses, and I ended up asking myself what would be the best short topic to teach and how to teach it.

Best short topic

There are two ways to answer the first question, one boring and one more interesting. The boring answer is that the course instructor selected the topic. The interesting one goes like this. I have taken short R courses before and taught others, and it’s always overwhelming for the students. You get to cover many concepts, get familiarized with R’s syntax, and in the end without lots of practice it’s very challenging to retain much of the information. I think that students love it when they learn how to do something simple that could be the first building block for many of their projects. In parallel, I think that one of the coolest R topics you can learn in an hour is how to create reproducible documents with rmarkdown (Allaire, Cheng, Xie, McPherson, et al., 2015).

Learning how to use a single function, render() in this case, is as simple as it gets. And using the RStudio Desktop is even simpler. Of course, it can easily get complicated. For example, on a new computer you need to install all the LaTeX dependencies if you want to create PDF files. That task can take some time and maybe scare away some new users. But PDF files are really a plus in this case since you can start creating HTML and Word documents. Other complications arise when a user is interested in more control over formatting the file, but like I said earlier, all you need is a simple building block and rmarkdown is clearly one of them.

This is why the final answer to the first question was teaching how to use rmarkdown to create reproducible reports (HTML, Word files) using R.

How to teach it

Teaching a short topic to a beginner’s audience is no easy feat. In the past I’ve made lectures that have the code for every single step and many links to resources where students can learn some details. That is, I’ve created the lectures in such a way that a student can later use them as reference and follow them without an instructor explaining them.

That’s a strategy that I think works on the long run. However, it makes the actual lecture boring and very limited in interactivity. At the JHSPH biostat computing club, other students have chosen to use a lot of images, funny to witty quotes, and asked listeners to voice their opinions. I’ve come to enjoy those presentations and I decided to create my lecture following that trend.

I started off with a series of questions about reproducible research and asked students to voice their opinions and to define a few key concepts. A couple were aware of the difference between reproducibility and replicability, but most were not. I also questioned them and presented them verbally with some famous cases, so they could realize that it’s a fairly complicated matter. Next I presented some answers and definitions from the Implementing Reproducible Research book.

Specifically talking about R, I showed the students several documents I’ve created in the past and asked whether they thought that they could reproduce the results or not. Basically, I wanted to highlight that when using R, you really need the session information if you want to reproduce something. Specially if the analysis involves packages under heavy development.

After having motivating the need for reproducible documents, I briefly showed what rmarkdown is with some images from RStudio shown below.

That gave the students a general idea of how these documents look when you are writing them. But the most important part was showing them examples of how the resulting documents look like. That is, I showed them some complicated projects so they could imagine doing one themselves. The examples included some books, but given the audience I think that the one that motivated them most was Alyssa Frazee’s polyester reproducible paper (check the source here). I also showed them some of the cool stuff you can create with HTML documents: basically adding interactive elements.

From there, we left the presentation and I demo’ed how to use RStudio to write rmarkdown documents, the Markdown syntax, where to find help, etc.

By this point, I think the lecture was quite complete and the students were motivated. However, from my past experience, I’ve come to realize that students will easily forget a topic if they don’t practice doing it. That is why even before making the lecture I spent quite a bit of time designing two practice labs. Both labs involved creating a rmarkdown document.

The first lab included some cool illusion plots which involved a lot of R code. The code wasn’t the point, but simply learning some of the basics such as what is a code chunk, some of Markdown’s syntax, specifying some code chunk options, adding the session information, and using inline R code to show the date when the document was made. Ahh, and of course, uploading your HTML document to RPubs (see mine). I know that not everyone is a fan of RPubs, but I imagined that students would get super excited that they made something that they could then show their colleagues and friends. And some did!

Sadly, we didn’t have enough time for the second lab. I did explain to the students what it was about, but they didn’t have time to do it themselves. For this second document, I wanted the students to learn how to create a document reporting some results where all the numbers in the text are written by R instead of copy-pasting them.

Conclusions

As you can see, I enjoyed thinking what to teach and specially how to teach a short topic to beginner R students. Thanks to having one of the later sessions, I could teach them how to use rmarkdown in a way that hopefully left them highly motivated to do it themselves. I hope that most of them will take that they learned in that module and others and apply them in their day to day work.

References

You can find the lecture itself here but like I said earlier, it was designed for class and not for being used as a reference. However, the lab and it’s key might be more useful.

Citations made with knitcitations (Boettiger, 2015).

[1] J. Allaire, J. Cheng, Y. Xie, J. McPherson, et al. rmarkdown: Dynamic Documents for R. R package version 0.7. 2015. URL: http://CRAN.R-project.org/package=rmarkdown.

[2] C. Boettiger. knitcitations: Citations for Knitr Markdown Files. R package version 1.0.6. 2015. URL: http://CRAN.R-project.org/package=knitcitations.

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

Use hidden advanced arguments for user-friendly functions

Thu, 11 Dec 2014 00:00:00 +0000

As a user

Imagine that you are starting to learn how to use a specific R package, lets call it foo. You will look at the vignette (if there is one), use help(package = foo), or look at the reference manual (for example, devtools’ ref man). Eventually, you will open the help page for the function(s) you are interested in using.

?function_I_want_to_use

In many packages, there is a main use case that is addressed by the package. A common strategy is to export a main function. That function will likely have a long list of arguments. So as a new user, you are suddenly exposed to a complicated help page and you will want to figure out which arguments you need to use.

As a developer

From the developer’s side, you want to give users control over several details. Each detail you want the user to control involves one more argument in your function. Sooner rather than later, you will have a long list of arguments. This increases the learning curve for new users of your package, and can potentially scare them away. That is contradictory of another goal you have as a developer: you want to get people to use your package.

Lets say that you are developing the function use_me(). If the details you want the users to control are actually arguments of other functions used inside use_me(), then you can simplify your function by using the ... argument. This argument is very well explained at The three-dots construct in R (Burns, 2013). It is very useful and can greatly simplify your life as a developer. Plus, it reduces the length of your help pages, thus making your package more user friendly.

However, if some of the details in use_me() are not arguments to other functions, then the common strategy is to write two functions. One is a low level function with arguments for all the details which might or might not export. Then, you write a second function that is a wrapper for the low level function and pre-specifies values for all the details. See the next minimal example:

# Don't export this function
.use_me <- function(arg1, arg2, verbose = TRUE) {
    if(verbose) message(paste(Sys.time(), 'working'))
    pmax(arg1, arg2)
}

#' @export
use_me <- function(arg1, ...) {
    .use_me(arg1, 0, ...)
}

## Lets see it in action
use_me(1:3)

## 2014-12-11 17:03:32 working

## [1] 1 2 3

use_me(-1:1, verbose = FALSE)

## [1] 0 0 1

In this example, the help page for use_me() is fairly short and friendly. You don’t expect users to be interested in changing arg2 much. Surely you could make it so the non-exported function .use_me() sets a default value for arg2.

Another strategy is to specify inside use_me() the default values for all the arguments you want to use while keeping the list of visible arguments short. That is, maintain the user friendliness of your functions while also giving them control over all the details. That is what you can do using dots() from dots (Collado-Torres, 2014). dots() is a very simple function that checks if ... has a specific argument, and if absent, it returns a default value. It can be seen in action below:

library('dots')
use_me_dots <- function(arg1, ...) {
    ## Default hidden arguments
    arg2 <- dots(name = 'arg2', value = 0, ...)
    verbose <- dots('verbose', TRUE, ...)
    
    ## Regular code
    if(verbose) message(paste(Sys.time(), 'working'))
    pmax(arg1, arg2)
}
use_me_dots(1:3)

## 2014-12-11 17:03:32 working

## [1] 1 2 3

use_me_dots(-1:1, verbose = FALSE)

## [1] 0 0 1

use_me_dots(-1:1, verbose = FALSE, arg2 = 5)

## [1] 5 5 5

dots is my solution to the problem of keeping functions user friendly while giving them control over all the details. The idea is that experienced users will be able to find what the advanced arguments are. While they could find them from the code itself, I do recommend describing the advanced arguments in a vignette targeted for these users.

Complications

Now, while ... is great, you might run into problems when use_me() calls two functions that have different arguments and that don’t have the ... argument. Such a scenario is illustrated below.

status <- function(arg3, status = TRUE) {
    if(status) print(arg3)
    return(invisible(NULL))
}
use_me_again <- function(arg1, ...) {
    res <- .use_me(arg1, 0, ...)
    status(res, ...)
    return(res)
}

## Seems to work
x <- use_me_again(1)

## 2014-12-11 17:03:32 working

## [1] 1

## But nope, it doesn't
use_me_again(1, verbose = FALSE, status = FALSE)

## Error in .use_me(arg1, 0, ...): unused argument (status = FALSE)

This scenario can happen when you are using functions from other packages. It’s happened to me in cases where the main function does have a ... argument but uses several internal functions that don’t use it.

In such situations, you might want to use formal_call() from dots. It figures out which are the arguments formally used by the function of interest and drops out un-used arguments from ..., thus avoiding this type of problem.

use_me_fixed <- function(arg1, ...) {
    res <- formalCall(.use_me, arg1 = arg1, arg2 = 0, ...)
    formal_call(status, arg3 = res, ...)
    return(res)
}

## Works now!
use_me_fixed(1, verbose = FALSE, status = FALSE)

## [1] 1

For a more complicated example, see the dots complex example in the vignette.

Conclusions

As a developer, it is possible to keep your functions user friendly while giving experienced users the option to control the fine tuning arguments which you don’t expect most users will want to tweak. My solution to this problem is implemented in dots (check it’s vignette). I’d love to hear what you think about it! I am specially interested on what users think about the idea of hidden advanced arguments (documented in an advanced users vignette).

I might try to get dots into a repository: probably in Bioconductor since most of the dots code was first implemented for derfinder.

PS I just found a similar function to dots(). It’s berryFunctions::owa() and you can find its code here.

References

Citations made with knitcitations (Boettiger, 2014).

[1] C. Boettiger. knitcitations: Citations for knitr markdown files. R package version 1.0.4. 2014. URL: https://github.com/cboettig/knitcitations.

[2] P. Burns. The three-dots construct in R - Burns Statistics. http://www.burns-stat.com/the-three-dots-construct-in-r/. 2013. URL: http://www.burns-stat.com/the-three-dots-construct-in-r/.

[3] L. Collado-Torres. dots: Simplifying function calls. R package version 1.0.0. 2014. URL: https://github.com/lcolladotor/dots.

Reproducibility

## Reproducibility info
library('devtools')
options(width = 120)
session_info()

## Session info-----------------------------------------------------------------------------------------------------------

##  setting  value                                             
##  version  R Under development (unstable) (2014-11-01 r66923)
##  system   x86_64, darwin10.8.0                              
##  ui       X11                                               
##  language (EN)                                              
##  collate  en_US.UTF-8                                       
##  tz       America/New_York

## Packages---------------------------------------------------------------------------------------------------------------

##  package       * version  date       source                                 
##  bibtex          0.3.6    2013-07-29 CRAN (R 3.2.0)                         
##  devtools      * 1.6.1    2014-10-07 CRAN (R 3.2.0)                         
##  digest          0.6.4    2013-12-03 CRAN (R 3.2.0)                         
##  dots          * 1.0.0    2014-11-14 Github (lcolladotor/dots@a933540)      
##  evaluate        0.5.5    2014-04-29 CRAN (R 3.2.0)                         
##  formatR         1.0      2014-08-25 CRAN (R 3.2.0)                         
##  httr            0.5      2014-09-02 CRAN (R 3.2.0)                         
##  knitcitations * 1.0.4    2014-11-03 Github (cboettig/knitcitations@508de74)
##  knitr         * 1.7      2014-10-13 CRAN (R 3.2.0)                         
##  lubridate       1.3.3    2013-12-31 CRAN (R 3.2.0)                         
##  memoise         0.2.1    2014-04-22 CRAN (R 3.2.0)                         
##  plyr            1.8.1    2014-02-26 CRAN (R 3.2.0)                         
##  RColorBrewer  * 1.0.5    2011-06-17 CRAN (R 3.2.0)                         
##  Rcpp            0.11.3   2014-09-29 CRAN (R 3.2.0)                         
##  RCurl           1.95.4.3 2014-07-29 CRAN (R 3.2.0)                         
##  RefManageR      0.8.40   2014-10-29 CRAN (R 3.2.0)                         
##  RJSONIO         1.3.0    2014-07-28 CRAN (R 3.2.0)                         
##  rstudioapi      0.1      2014-03-27 CRAN (R 3.2.0)                         
##  stringr         0.6.2    2012-12-06 CRAN (R 3.2.0)                         
##  XML             3.98.1.1 2013-06-20 CRAN (R 3.2.0)

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

I wrote dots a month ago and the post itself today during our bi-weekly blog meeting.

An xpd-tion into R plot margins

Fri, 21 Nov 2014 00:00:00 +0000

This is a guest post by Prasad Patil that answers the question: how to put a shape in the margin of an R plot?

The help page for R’s par() function is a somewhat impenetrable list of abbreviations that allow you to manipulate anything and everything in the plotting device. You may have used this function in the past to create an array of plots (using mfrow or mfcol) or to set margins (mar or mai).

Way down toward the end of the list is the often-overlooked xpd parameter. This value specifies where in the plotting device an object can actually be plotted. The default is xpd = FALSE, which means that plotting is clipped, or restricted, to the plotting region. In other words, if your plot has xlim = c(0, 10) and ylim = c(0, 10) and you try to plot the point (-1, -1), it will not appear anywhere in the device.

xpd takes two other values, TRUE and NA, which limit plotting to the figure and device region, respectively. If you’re fuzzy on plotting terms, this tutorial presents those topics well.

Plotting outside the plot

If you want to plot outside of the plotting region, I find that setting xpd = NA easiest since it opens up all external space. We also need to make sure that we keep space outside of the plot so that we have room to place our objects. Let’s say we want to put an ugly border above and below our plot:

# Set xpd=NA and expand the top and bottom margins
par(xpd = NA, mar = par()$mar + c(2.5, 0, 1, 0))
plot(1:10)
# Note that the rectangle we make here has corner coordinates outside of
# our plotting device
rect(-5, 11, 12, 14, col="red")
# Random dots in our rectangluar region
points(runif(100, -4.2, 12.8), runif(100, 11.2, 13.6), col = "green", pch = 19, cex = 1.2)
# And another rectangle for below
rect(-5, -1.7, 12, -3.5, col="red")
points(runif(100, -4.2, 12.8), runif(100, -3.3, -1.8), col = "green", pch = 19, cex = 1.2)

Here we mentally extend the axes of our plot to determine where to put our margin elements. One can imagine a diagonal for the top rectangle running from (-5,11) to (12,14). Neither of these points appear in the plot itself, but we used the established axes to estimate them and plot outside the plotting region.

Images outside the plot

Now let’s say we want to add a logo or other external image in the margin of our plot. We will use R’s png library to load a PNG image and rasterImage() to plot it:

## If needed: install.packages("png")
library(png)
img <- readPNG("logo.png")
par(xpd = NA, mar=par()$mar + c(3, 0, 0, 0))
plot(1:10)
rasterImage(img, 0.5, -2.5, 10.5, -1)

Here we used the png library and the r asterImage() command to read in and plot the “logo.png” file. Based on the previously-known dimensions of the logo, we can choose which points to use as endpoints for the image. Note that this image may appear stretched or contorted depending on the size of your R plot device, and it will not stay consistent if you resize.

Where do I start using Bioconductor?

Thu, 16 Oct 2014 00:00:00 +0000

I was recently asked where do I get started with Bioconductor? and thought this would be a good short post.

What is BioC?

Briefly, Bioconductor (Gentleman, Carey, Bates, and others, 2004) is an open source project that hosts a wide range of tools for analyzing biological data with R (R Core Team, 2014). These analysis tools are bundled into packages which are designed to answer specific questions or to provide key infrastructure. If this sounds like something you are interested in, visit bioconductor.org.

Obviously, you need to know the basics about R in order to use Bioconductor.

Getting started

bioconductor.org has a section in it’s front page titled get started with Bioconductor. There you will find links that explain how to install it or to explore the available packages.

You have a use case

If you have a particular use case in mind, I recommend browsing the software packages and searching for some key words. For example, you might be interested in high throughput sequencing of RNAs and if you search RNAseq or RNA-seq you can find a good set of packages to start. Alternatively, use the biocViews tree menu to explore specific categories of packages.

Once you find a set of packages that have descriptions that appeal to you, explore their vignettes. These are either PDF or HTML documents that explain what the package does to new users. They also exemplify how to tie together the different functions in the package, which is a key piece of information. For example, in the RNA-seq example you will find the DEXSeq package. DEXseq (Anders, Reyes, and Huber, 2012) has a vignette called Analyzing RNA-seq data for differential exon usage with the “DEXSeq” package and from the page of the package you can access the PDF vignette.

Then it’s just a matter of exploring other packages, checking the vignettes and learning as you go.

You don’t have a use case

If you don’t have a specific use case in mind, it might pay off to start by exploring the Bioconductor workflows. These documents explain how to use different packages to accomplish specific type of analyses. They are great to learn what you can do with Bioconductor!

Another option is to look at the previous courses. For example, under the 2008 courses you’ll find to the course R/Bioconductor Curso Intensivo (Spanish) which I taught back in the day. As much as I would like to self promote myself, the best starting point is the most recent BioC20XX course: BioC2014. It has slides showcasing some of the newest packages and tutorials on how to use them.

An alternative is to look at some of the Bioconductor publications which includes books about Bioconductor and research papers describing some of the packages.

Once you find a set of packages that catch your eye, go look at their vignettes just like I explained in the you have a use case scenario.

Help tips

It’s not a matter of whether you will need help learning how to use Bioconductor. It’s just a matter of when. So don’t feel bad about having to ask for help!!

The very first place to start is to look at bioconductor.org at the Help section in the bottom. For example, you can find youtube videos contributed under the community section. There you can also find links to other blog posts explaining how to use Bioconductor. Take a peak at the other sections under Help before using the Bioconductor support site: it’s where you can ask very specific questions and interact with the maintainers of the packages you are using.

Finally, if you are interested in new developments, then check the latest newsletter, for example the October 2014 one.

Good luck using Bioconductor!

References

Citations made with knitcitations (Boettiger, 2014).

[1] S. Anders, A. Reyes and W. Huber. “Detecting differential usage of exons from RNA-seq data.” In: Genome Research 22 (2012), p. 4025. DOI: 10.1101/gr.133744.111.

[2] C. Boettiger. knitcitations: Citations for knitr markdown files. R package version 1.0.2. 2014. URL: https://github.com/cboettig/knitcitations.

[3] R. C. Gentleman, V. J. Carey, D. M. Bates and others. “Bioconductor: Open software development for computational biology and bioinformatics”. In: Genome Biology 5 (2004), p. R80. URL: http://genomebiology.com/2004/5/10/R80.

[4] R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. Vienna, Austria, 2014. URL: http://www.R-project.org/.

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

Concerns that can deter potential orders for developing Shiny apps

Tue, 10 Jun 2014 00:00:00 +0000

A few weeks ago I was invited to a meeting where a group was interested in exploring options for replacing their contract with a propriety software. They invited me because they saw some resemblances between a Shiny application I made and the features they need. It is a relatively small project and it seemed feasible to implement, but well, some details could have been tricky to code. During the meeting I explained what Shiny is, showcased some of the Shiny apps I’ve made, and proposed some options including a simple site password.

A few days after the meeting, they raised three concerns.

Privacy and security of their data. Specially with distribution of the site password.
Technical support. They wanted something more than community support.
Continuity of the project. Specially since their team might lack the technical expertise to support and modify the Shiny app after it is built.

These are all valid concerns and they are not something I have dealt with or been concerned with other Shiny apps I have deployed. Thus, I ended up finding more information and writing up a long reply which I am modifying into a post format here. If you identify other ways to approach these concerns that I missed, share the knowledge please!

Shiny overview

Shiny is an R package that allows creating web applications with R. A user opens the app on their browser, specifies a given set of inputs, these are transmitted to an R session, some code is evaluated with the input options, output is produced and transmitted back to the browser. Because it is so easy to use and useful, it has been very popular. That is the gist of it!

Deploying a Shiny app

Once you develop a Shiny app, you have to deploy it on a server so users can access it through their browsers. Here I describe several options to do so.

The application itself needs “Shiny Server” to run. An implementation of Shiny Server is free via ShinyApps (option 1) or if you have your own server you can install the open source version of Shiny Server (option 2). Alternatively, you can pay for Shiny Server Pro and also install it in your own server (option 3). Note that glimmer.rstudio.com is previous version of ShinyApps and you can no longer get accounts on this server.

Option 1 has the advantage of being completely free and that there is no need to pay for your own server. It does rely on ShinyApps being online, which should not be a main problem since it is hosted on the cloud. Meaning that it is supposed to be robust.

Option 2 has the advantage of having technical support for the server (not the app) from whoever is providing the server (could be a company or the school). However, you need to have someone capable of installing Shiny Server and updating it whenever it’s necessary to do so.

Option 3 allows you to have high level password security (via SSL) to any Shiny app you host. Plus RStudio provides technical support for Shiny. The server technical support would still have to come from whoever is providing the server. The main disadvantage is that this option is very expensive (even with the academic pricing discount).

Here is some information you might be interested on checking about Shiny:

Concerns

Privacy

Privacy issues in terms of password sharing can be limited by changing the password frequently. Privacy in terms of protection versus hacking is a different subject and I could implement something like this demo (username: withr, password: 12345678) as described in this blog post. However, if you want further protection vs hacking, you would need Shiny Server Pro. Here is a demo of the security provided by Shiny Server Pro (with multiple users too). I understand that hacking was not the main concern you had, but well, as I said password-sharing can be mitigated by frequent changes to the app site.

Technical support

I mostly answered the different options regarding technical support when I described the three options for deploying a Shiny app. As for getting support from IT, they would need to learn how to use Shiny.

Continuity

Shiny is one of the most important projects for RStudio so they invest a lot in it, the community provides great answers very quickly, and a lot of people are learning how to use it. If you need an R programmer later on instead of a student, you could describe the job at R Users and employ someone that way. As far as the Biostats department is concerned, I know that several students are using Shiny and some have even published Shiny Apps in the academic literature. Here are some examples:

interAdapt is a Shiny app written by Aaron Fisher (Biostats student) and published here.
shinymethyl, a shiny application developed by Jean-Philippe Fortin described here with a demo here.
committees is a Shiny app made by Alyssa Frazee for verifying your school oral and/or defense committees. It’s described in this post.
ENAR 2014 is a Shiny app made by John Muschelli for the ENAR2014 conference which allowed you to quickly identify talks you might be interested in. It’s described in this post.

Branding

Regarding branding, a Shiny App can be embedded on a website. For illustrating this, I embedded two apps I made here.

Conclusions

In the end, this group decided to use something else instead of a Shiny app for now. I do not know exactly why they made that decision, but I can see that their concerns are probably shared with other potential Shiny users. These concerns are a challenge RStudio must be dealing with and I am curious to know how they would have approached them.

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

This post is part of the summer 2014 Bmore Biostats’ iron blogger challenge.

Simple mortgage calculator

Tue, 22 Apr 2014 00:00:00 +0000

Have you ever thought of borrowing some money? A common case is when you have to borrow money for buying a house, which is called a mortgage. Wikipedia (‘Mortgage loan’ entry) goes into much more detail about the definition than what I’ll cover.

One of the aspects you have to evaluate when considering a loan such as a mortgage is how much interest you will be charged and how long the loan will be. Those two determine your monthly payment.

Comparing loans

That is, given the loan amount, you can compare different loans that offer different rates and/or loan lengths. There are many tools out there that allow you to do so and are generally called mortgage calculators. Although the more detailed term is amortization calculator. From wikipedia(‘Amortization calculator’ entry) we find the so called annuity formula which helps you calculate your monthly payment.

Several such mortgage calculators have been features in R Bloggers before such as (C, 2010) and (BioStatMatt). Plus there are many other alternatives on the web.

My use case

However, a month or so ago I got curious about comparing some loans which had a different twist than normal. Given some circumstances, I wanted to compare some loans where the loaner is willing to receive payments every few months yet with interests being compounded (that is, calculated) on a different time frame. Obviously, the easiest would be to compound interest at the same frequency as payments are made. Plus, probably most banks would not offer such flexible loans.

Digging around, I did find modifications to the annuity formula that allow non-monthly payments (Compound Interest Formula) and (What is compound interest?).

shiny app

Being interested in R code, shiny (RStudio Inc.) apps and trying out integrating rCharts (Vaidyanathan, 2013) with shiny, I ended up coding my own mortgage calculator which you can find here (Simple Mortage Calculator). The code is publicly available at GitHub.

It has a couple of simple inputs:

Amount to borrow
Interest rate
Loan duration
Payment frequency
How frequently interests are compounded
Month of the first payment

The option to begin the loan now but accept the first payment much later is there also because of the complicated use case I had in mind. Again, that is a feature a bank will most likely not offer. But it’s something I needed to take into consideration.

Going back to the app, I tried explaining all the inputs as much as possible. The output is relatively straight forward.

First, there is a line with focus D3 interactive plot (Line Chart with View Finder) which shows the principal (the amount you owe) over months as well as how much you’ve paid already. The bottom panel allows you to zoom into a specific time range as shown below.

Next, the information is shown in more detail as an interactive data table (DataTables) 12 months at a time. The interactive part makes it very easy to search for a specific month and look at the state of the loan.

Finally, you can download the amortization table in CSV format for your records.

PS You can also access the app via shinyapps at this url. It is the first time I’ve deployed an app there as I wanted to test shinyapps out.

Conclusions

The experience of coding the shiny app (Simple Mortage Calculator) was interesting as I did learn a couple of new things. The same was true for figuring out the calculations for the more flexible options.

Finally, but not least, the shiny app was useful for my use case and was informative for comparing different loan options. Hopefully it will be useful for other users!

References

Citations made with knitcitations (Boettiger, 2014).

Mortgage loan - Wikipedia, the free encyclopedia. Wikipedia http://en.wikipedia.org/wiki/Mortgage_loan
Amortization calculator - Wikipedia, the free encyclopedia. Wikipedia http://en.wikipedia.org/wiki/Amortization_calculator
C, (2010) Mortgage Calculator (and Amortization Charts) with R. R-Chart http://www.r-chart.com/2010/11/mortgage-calculator-and-amortization.html
BioStatMatt, Mortgage Refinance Calculator | BioStatMatt. BioStatMatt http://biostatmatt.com/archives/1908
Compound Interest Calculator. http://www.calculator.net/compound-interest-calculator.html
Compound Interest. http://math.about.com/library/weekly/aa042002a.htm
Simple Mortgage Calculator. http://glimmer.rstudio.com/lcolladotor/mortgage/
Novus Partners, Line Chart With View Finder - NVD3. http://nvd3.org//examples/lineWithFocus.html
DataTables (table plug-in for jQuery). DataTables (jQuery plug-in) https://datatables.net/
Carl Boettiger, (2014) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations
Ramnath Vaidyanathan, (2013) rCharts: Interactive Charts using Polycharts.js.
RStudio , Inc. , (2014) shiny: Web Application Framework for R. http://CRAN.R-project.org/package=shiny

Reproducibility

sessionInfo()

## R version 3.1.0 (2014-04-10)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitcitations_0.5-0 bibtex_0.3-6        knitr_1.5.27       
## 
## loaded via a namespace (and not attached):
##  [1] codetools_0.2-8 digest_0.6.4    evaluate_0.5.3  formatR_0.10   
##  [5] httr_0.3        RCurl_1.95-4.1  stringr_0.6.2   tools_3.1.0    
##  [9] XML_3.98-1.1    xtable_1.7-3

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

rMaps Mexico map

Wed, 26 Feb 2014 00:00:00 +0000

It’s exciting when great people help each other get things done

This is a simple networking story, which might not be surprising to some but I was happily surprised by it. This is how the story goes:

Two weeks ago rMaps (Vaidyanathan, 2014) was released. After making a blog post about it I thought about using it to make a map of the homicide rate in Mexico over the recent years. First, I had the question of how to make custom maps with rMaps. @tyokota had the same question and started asking Ramnath about it in rMaps issue 6. Then I realized I needed a specific file with the map information. Google lead me to @diegovalle who has created the map from official Mexican sources, downloaded the homicide data, cleaned it, and made several maps and analyses: all his work is very impressive! I thought that it’d be very cool if @diegovalle and Ramnath connected, and they did! I saw them interacting via Twitter ( here and here) and via GitHub. After sharing @diegovalle’s work with my friends, it turned out that some old friends already knew him ( here and high school friends). Another friend was interested in additional features and I suggested her to contact @diegovalle via Twitter: he quickly replied as you can see here.

Beyond how impressive rMaps and @diegovalle’s work on mexican data are, I was amazed by the willingness to help each other and how great people easily connected. I believe this is one of the great features of both GitHub and Twitter where you can share your code, ask questions, try to answer them, meet people working with your tools, etc. You can even offer to PayPal a beer like @tyokota did.

After all their great work, now someone like me (aka, without knowing javascript, Datamaps, etc) can walk you through an example of making an interactive choropleth map showing the homicides rates in Mexico from 1997 to 2013.

Homicides rates in Mexico 1997-2013

The first thing we need to make a custom map using rMaps is a topojson file which in this case specifies the mexican states boundaries. This process is explained in more detail by @tyokota at custom-map which you can view here.

In this particular example, INEGI which is the National Institute of Statistics and Geography of Mexico has a map of the mexican states. @diegovalle explained how to download it here.

But before doing so, you might to install topojson like I did below following the installation instructions. In the terminal:


## Install node.js following instructions at https://github.com/mbostock/topojson/wiki/Installation
brew install node
## Install topojson
npm install -g topojson

## Download map info from INEGI (Mexican official source)
curl -o estados.zip http://mapserver.inegi.org.mx/MGN/mge2010v5_0.zip
## Decompress file
unzip estados.zip
## Create shapefile
ogr2ogr states.shp Entidades_2010_5.shp -t_srs "+proj=longlat +ellps=WGS84 +no_defs +towgs84=0,0,0"
## id-property needed so that DataMaps knows how to color the map
topojson -o mx_states.json -s 1e-7 -q 1e5 states.shp -p state_code=+CVE_ENT,name=NOM_ENT --id-property NOM_ENT

Now that we have the topojson file mx_states.json we need to get the actual homicide data. @diegovalle has gone through the whole process of acquiring the data from official mexican sources and cleaning it. Lets download it.

# Download crime data
## From crimenmexico.diegovalle.net/en/csv
## All local crimes at the state level
download.file("http://crimenmexico.diegovalle.net/en/csv/fuero-comun-estados.csv.gz", 
	"fuero-comun-estados.csv.gz")

The data is not completely ready for us to use it and we need to reshape it a bit. In particular, we want to consider only the intentional homicides and group the data by state and date. We can get this to work by using dplyr (Wickham & Francois, 2014).

## Load required packages
library("dplyr")

## Load the crime data
crime <- read.csv("fuero-comun-estados.csv.gz")

## Only intentional homicides
crime <- subset(crime, crime == "HOMICIDIOS" & type == "DOLOSOS")

## Sum homicides by firearm, etc and group by state and date
hom <- crime %.%
  filter(year %in% 1997:2013) %.%
  group_by(state_code, year, type) %.%
  summarise(total = sum(count, na.rm = TRUE),
            population = mean(population) ) %.%
  mutate(rate = total / population * 10^5) %.%
  arrange(state_code, year)

## How are states coded?
summary(hom$state_code)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    1.00    8.75   16.50   16.50   24.20   32.00

We have the slight inconvenience that states are coded as integers from 1 to 32 instead of using their names. Using another of the files supplied by @diegovalle we can merge the codes. This requires using the foreign (R Core Team) package for loading a dbf file and then merging both data sets with plyr (Wickham, 2011).

## Needed for read.dbf
library("foreign")

## The dbf from the state shapefile needed to merge state_code with state
## names
codes <- read.dbf("states.dbf")
codes$NOM_ENT <- iconv(codes$NOM_ENT, "windows-1252", "utf-8")
codes$CVE_ENT <- as.numeric(codes$CVE_ENT)
codes$OID <- NULL
names(codes) <- c("state_code", "name")

## Load plyr for join(). Loading it before creates a problem with the dplyr
## call above
library("plyr")

## Names needed for the map
hom <- join(hom, codes)

## Lets look at the data
head(hom)

##   state_code year    type total population   rate           name
## 1          1 1997 DOLOSOS   355     958126 37.051 Aguascalientes
## 2          1 1998 DOLOSOS    66     975585  6.765 Aguascalientes
## 3          1 1999 DOLOSOS    27     992515  2.720 Aguascalientes
## 4          1 2000 DOLOSOS    14    1009215  1.387 Aguascalientes
## 5          1 2001 DOLOSOS    22    1026437  2.143 Aguascalientes
## 6          1 2002 DOLOSOS    26    1044578  2.489 Aguascalientes

Great! We now have state names under name and the intentional homicide rate under rate (in homicides per 100,000) for each specific year. We can thus proceed to making the interactive choropleth map using the ichoropleth function described by Ramnath here. This requires specifying the topojson file which is specified via dataUrl, the name of the map specified via scope and the most tricky part (for me at least) is that we need to specify the setProjection. These are all properties of the Datamaps library. In particular, the wiki describes how to use custom maps but this requires some javascript knowledge.

## Make the map
library("rMaps")
d1 <- ichoropleth(rate ~ name, data = hom, ncuts = 9, pal = 'YlOrRd', 
    animate = 'year',  map = 'states'
)
## Note that I am hosting the mx_states.json in Dropbox
## but if you are doing it locally, you only need
## dataUrl = "mx_states.json"
d1$set(
  geographyConfig = list(
   dataUrl = "https://dl.dropboxusercontent.com/u/10794332/mx_states.json"
  ),
 scope = 'states',
 setProjection = '#! function( element, options ) {
   var projection, path;
   projection = d3.geo.mercator()
    .center([-89, 21]).scale(element.offsetWidth)
    .translate([element.offsetWidth / 2, element.offsetHeight / 2]);

   path = d3.geo.path().projection( projection );
   return {path: path, projection: projection};
  } !#'
)
d1$save('rMaps.html', cdn = TRUE)

The end result is shown below:

You can also share the map using the publish method as shown below:

d1$publish("Intentional homicides rates for Mexico 1997-2013")
## You'll need a GitHub account

You will get a link to the rCharts viewer such as this one or if you prefer, you can also view the result using Pagist as shown here.

The code presented in this post was written by @diegovalle which can you view here and Ramnath which is shown here. I also figured out the trick of hosting the topojson file at Dropbox from @tyokota’s code as I was running into Access-Control-Allow-Origin errors when hosting it in my academic website. Finally, but not least, Ramnath appropriately insists that all of this would not be possible without libraries such as Datamaps.

References

Citations made with knitcitations (Boettiger, 2014).

Hadley Wickham, Romain Francois, (2014) dplyr: dplyr: a grammar of data manipulation. http://CRAN.R-project.org/package=dplyr
R Core Team , (2014) foreign: Read Data Stored by Minitab, S, SAS, SPSS, Stata, Systat, Weka, dBase, …. http://CRAN.R-project.org/package=foreign
Carl Boettiger, (2014) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations
Hadley Wickham, (2011) The Split-Apply-Combine Strategy for Data Analysis. Journal of Statistical Software 40 (1) 1-29 http://www.jstatsoft.org/v40/i01/
Ramnath Vaidyanathan, (2014) rMaps: Interactive Maps from R.

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] rMaps_0.1.1         plyr_1.8            foreign_0.8-59     
## [4] dplyr_0.1.1         knitcitations_0.5-0 bibtex_0.3-6       
## [7] knitr_1.5          
## 
## loaded via a namespace (and not attached):
##  [1] assertthat_0.1     digest_0.6.4       evaluate_0.5.1    
##  [4] formatR_0.10       grid_3.0.2         httr_0.2          
##  [7] lattice_0.20-24    rCharts_0.4.2      RColorBrewer_1.0-5
## [10] Rcpp_0.11.0        RCurl_1.95-4.1     RJSONIO_1.0-3     
## [13] stringr_0.6.2      tools_3.0.2        whisker_0.3-2     
## [16] XML_3.95-0.2       xtable_1.7-1       yaml_2.1.10

Check other topics on #rstats.

Automatically coloring your R output in the terminal using colorout

Mon, 17 Feb 2014 00:00:00 +0000

Thanks to Alyssa Frazee I just learned about the colorout package (Aquino, 2013). It modifies R so that the output is in different colors, making it much more pleasant to use R in the terminal.

Do note that colorout is not available from CRAN, but you can easily install by following the instructions on the colorout site (Official site) reproduced below:

download.file("http://www.lepem.ufc.br/jaa/colorout_1.0-2.tar.gz", destfile = "colorout_1.0-2.tar.gz")
install.packages("colorout_1.0-2.tar.gz", type = "source", repos = NULL)

The next step is to then load colorout automatically when I start R. The problem is that I don’t use R solely on the terminal. I easily figured out how to do so thanks to the error message you get when attempting to load colorout on the R GUI. I thus ended up adding the following lines to my .Rprofile (both locally and in the cluster):

## Change colors when running R in the terminal
if (Sys.getenv("TERM") == "xterm-256color") library("colorout")

Now I have pretty R output in the terminal! Thanks again Alyssa! See her original tweet below:

problem: R output in Terminal isn't colorful. SOLUTION: http://t.co/Vd6OoRoUU5
— Alyssa Frazee (@acfrazee) February 17, 2014

colorout has been around for a while and was even at CRAN for some time. I guess that I’m just late to the party.

If the default colorout colors don’t work for you, check functions such as setOutputColors256. This post shows how you can do it and includes screenshots of the output. Other package details are included here and here.

References

Citations made with knitcitations (Boettiger, 2014).

Jakson Aquino, (2013) colorout: Colorize R output on terminal emulators. http://www.lepem.ufc.br/jaa/colorout.html
colorout. http://www.lepem.ufc.br/jaa/colorout.html
Carl Boettiger, (2014) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitcitations_0.5-0 bibtex_0.3-6        knitr_1.5          
## 
## loaded via a namespace (and not attached):
##  [1] codetools_0.2-8 digest_0.6.4    evaluate_0.5.1  formatR_0.10   
##  [5] httr_0.2        RCurl_1.95-4.1  stringr_0.6.2   tools_3.0.2    
##  [9] XML_3.95-0.2    xtable_1.7-1

Check other topics on #rstats.

rMaps released

Mon, 10 Feb 2014 00:00:00 +0000

Ramnath Vaidyanathan just released his new R interactive package, rMaps (Vaidyanathan, 2014). The packages relies on the development version of his widely known rCharts package (Vaidyanathan, 2013) as well as javascript libraries that specialize in maps. If you don’t know Ramnath, he is one of the most active R developers out there!! You can see that from his GitHub profile.

The package is very new and still under development, but I bet that Ramnath released it to get us users excited and maybe find some helpful hands to document it and further develop it.

I don’t know about you, but I surely got excited about the package from his intro video:

It’s a simple screen cast with good music.

In the GitHub rMaps repository you can find the simple installation instructions as well as three different examples. They all work if you run them in the latest version of RStudio otherwise you might run into a couple minor hiccups like I did.

Just to get you excited, this is the third example where you can easily add markers with pop ups.

suppressMessages(library("rMaps"))

map <- Leaflet$new()
map$setView(c(51.505, -0.09), zoom = 13)
map$tileLayer(provider = "Stamen.Watercolor")
map$marker(c(51.5, -0.09), bindPopup = "Hi. I am a popup")
map

You can view the interactive version of the example here – I’m sure that a feature will be added later to make it easy to share the maps you make.

Overall, I think that this is a great start and I look forward to using it. For now, don’t be discouraged with the lack of documentation. I’m sure that if you ask nicely Ramnath will answer asap!

References

Citations made with knitcitations (Boettiger, 2014).

Carl Boettiger, (2014) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations
Ramnath Vaidyanathan, (2013) rCharts: Interactive Charts using Polycharts.js.
Ramnath Vaidyanathan, (2014) rMaps: Interactive Maps from R.

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] rMaps_0.1           knitcitations_0.5-0 bibtex_0.3-6       
## [4] knitr_1.5          
## 
## loaded via a namespace (and not attached):
##  [1] codetools_0.2-8    digest_0.6.4       evaluate_0.5.1    
##  [4] formatR_0.10       grid_3.0.2         httr_0.2          
##  [7] lattice_0.20-24    plyr_1.8           rCharts_0.4.2     
## [10] RColorBrewer_1.0-5 RCurl_1.95-4.1     RJSONIO_1.0-3     
## [13] stringr_0.6.2      tools_3.0.2        whisker_0.3-2     
## [16] XML_3.95-0.2       xtable_1.7-1       yaml_2.1.10

Check other topics on #rstats.

How to upload files to Dropbox and Google Docs from R

Wed, 05 Feb 2014 00:00:00 +0000

Have you ever wondered whether you can upload files from R to Dropbox and/or Google Docs? I recently asked myself this question while making my most recent Shiny app (more later).

The answer is yes, you can upload files from R to these cloud services!

Dropbox

As far as I know, the best R package for uploading files to Dropbox is rDrop (Ram & Temple Lang, 2012). The whole setup is very well explained in it’s GitHub repository (Karthik). Basically you have to:

Install the package and it’s dependencies (some are only on GitHub).
Create a Dropbox app in your account.
Get the credentials info.
Authorize your access to the Dropbox app from R.
Save that info for later use.

dropbox_save()

Then uploading any type of R object to Dropbox becomes as easy as using the dropbox_save() function.

For example, lets save a simple vector with random data.

suppressMessages(library("rDrop"))

## Define credentials or load them if you defined them already
## dropbox_credentials <- dropbox_auth('Your app key', 'Your app secret')
load("dropbox_credentials.Rdata")

## Lets create some random data
set.seed(20140205)
x <- rnorm(1000)

## Lets check the args of the uploading function
args(dropbox_save)

## function (cred, ..., list = character(), file = stop("'file' must be specified"), 
##     envir = parent.frame(), precheck = TRUE, verbose = FALSE, 
##     curl = getCurlHandle(), ext = ".rda", .opts = list()) 
## NULL


## Then lets upload it to dropbox on the public folder
done <- dropbox_save(dropbox_credentials, x, file = "public/x")

## The result has some information, like the path of where you upload the
## file
names(done)

##  [1] "revision"     "rev"          "thumb_exists" "bytes"       
##  [5] "modified"     "client_mtime" "path"         "is_dir"      
##  [9] "icon"         "root"         "mime_type"    "size"

done$path

## [1] "/public/x.rda"

You can now actually download the “x.rda” file from here. That’s in case that you also wanted to share the file, and is obviously optional.

Note that you can get the link from withing R and don’t even need to use the Dropbox site.

dropbox_share(dropbox_credentials, file = "public/x.rda")

## $url
## [1] "https://db.tt/xzf3huXf"
## 
## $expires
## [1] "Tue, 01 Jan 2030 00:00:00 +0000"

dropbox_put()

What if you want to upload an actual file and not only R objects? That’s where dropbox_put() shines. Below is an example where we create an image, save it as a pdf file, and upload it to Dropbox.

## Lets create a sample file, in this case a pdf file
pdf("dropboxFig.pdf")
hist(x, freq = FALSE, col = "light blue")
tmp <- dev.off()

## Lets check the args for the uploading function
args(dropbox_put)

## function (cred, what, filename = what, curl = getCurlHandle(), 
##     ..., verbose = FALSE, contentType = "application/octet-stream") 
## NULL


## Now, lets upload the file
done <- dropbox_put(dropbox_credentials, what = "dropboxFig.pdf", filename = "public/dropboxFig.pdf")

## Again, the result contains some information about the file
names(done)

##  [1] "revision"     "rev"          "thumb_exists" "bytes"       
##  [5] "modified"     "client_mtime" "path"         "is_dir"      
##  [9] "icon"         "root"         "mime_type"    "size"

You can view the result here.

Google Docs

From what I found, it seems to me that RGoogleDocs (Temple Lang) is the package you’ll want to use for interacting with Google Docs from R. The manual (Temple Lang) explains all what you pretty much need to know. You should know though that you can only upload certain types of files.

For example, you can upload a text file as shown below.

suppressMessages(library("RGoogleDocs"))

## Load password
load("gpasswd.Rdata")

## Something to write
text <- "Hello world!\n"

## Authentificate
auth <- getGoogleAuth("lcolladotor@gmail.com", gpasswd)

## Connect to Google
con <- getGoogleDocsConnection(auth)

## Check the args for the uploading function
args(uploadDoc)

## function (content, con, name, type = as.character(findType(content)), 
##     binary = FALSE, asText = FALSE, folder = NULL, replace = TRUE, 
##     ...) 
## NULL


## Upload the file
done <- uploadDoc(content = text, con = con, name = "testFile.txt", type = "txt")

You can view the file here.

Conclusions

rDrop (Ram & Temple Lang, 2012) is very cool and easy to use. Compared to Google Docs, you have much greater flexibility on the type of files you can upload. I guess that will change in the future if there is a Google Drive from R package.

References

Citations made with knitcitations (Boettiger, 2014).

karthik, karthik/rDrop. GitHub https://github.com/karthik/rDrop
An simple R interface to Google Documents. http://www.omegahat.org/RGoogleDocs/run.html
Carl Boettiger, (2014) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations
Karthik Ram, Duncan Temple Lang, (2012) rDrop: Dropbox R interface.. http://github.com/karthikram/rDrop
Duncan Lang, RGoogleDocs: Primitive interface to Google Documents from R.

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] RGoogleDocs_0.7-0   rDrop_0.3-0         stringr_0.6.2      
##  [4] plyr_1.8            ROAuth_0.92.0       digest_0.6.4       
##  [7] RCurl_1.95-4.1      bitops_1.0-6        RJSONIO_1.0-3      
## [10] knitcitations_0.5-0 bibtex_0.3-6        knitr_1.5          
## 
## loaded via a namespace (and not attached):
## [1] evaluate_0.5.1 formatR_0.10   httr_0.2       tools_3.0.2   
## [5] XML_3.95-0.2   xtable_1.7-1

Check other topics on #rstats.

Creating awesome reports for multiple audiences using knitrBootstrap

Tue, 10 Dec 2013 00:00:00 +0000

As a biostatistics student, I use R very frequently when analyzing data. At the same time, I interact with other researchers, some who know how to use R (R crowd) and some who don’t (yet!): no-R crowd. This means that I have to be able to communicate my results to two crowds. It is important that I can quickly provide the code in case that the R savvy want to look at it: maybe they find a bug and report it ^^. Ideally I want to avoid having to write (organize, share, etc) two crowd-specific reports.

A solution to this problem is to create reproducible reports that contain the R code, the results, and interpretation. For the specific scenario I am talking about, reproducibility is a plus, however I believe that it is important for research; albeit not the topic of this post. One of the strongest packages out there to create such reports is knitr (Xie, 2013). It is specially easy to create Rmd files from which you can generate HTML reports. Then using RStudio you can share them via RPubs, a private folder on Dropbox, etc. From example, this is a presentation without the slide formatting I shared more than a year ago.

Using knitr (Xie, 2013) is definitely a step in the right direction. However, you soon find yourself desiring a better template. This is where knitrBootstrap (Hester, 2013) comes in. This package was initialized in March 20th, 2013 by Jim Hester and hosted on it’s GitHub repository. I was sold on the idea early on and I am now making this post in part as a tribute to celebrate that it has been available via CRAN for nearly 5 months now.

So what can you do with knitrBootstrap (Hester, 2013)? In my opinion, you get the ideal solution (or very close at the least) to the problem I described at the beginning. Basically, you get a HTML report that has the interpretation and results which is what the no-R crowd wants to read, and the R code easily available at the click of a button for the R crowd. In addition, the report is much more nicely formatted which is pleasant to the eye. Furthermore, a menu with the sections is included which is very useful when navigating the report and for jumping to specific sections. To save space, the plots are saved as thumbnails and you can click on them to get the full view. Finally, you can choose to display toggle menus for allowing the users to change the default text and code formatting.

How do you use this package? The main workhorse is the knit_boostrap() function. The initial arguments are similar to those you find in knitr::knit() while the new features are controlled using:

boot_style You can select out of 11 or so options for the default formatting. Basically, you choose one of the Bootstrap themes available.
code_style Similar to boot_style but for controlling the appearance of the code chunks.
chooser Allows you to control if you want a toggle menu so the user can choose (hence the name) the bootstrap and/or code styles.
thumbsize For controlling the size of figure thumbnails.
show_code Whether by default the code is shown. I set this to FALSE in order to get a report that by default is accessible for the no-R crowd. The R crowd can then click to see the code for each code chunk or use the menu on the bottom to show all the code at once.
show_output Similar to show_code but for controlling the visibility of the output produced from the code. I set this to TRUE as you normally want to show the output to both the no-R and R crowds.
show_figure Whether you want to show the plots or not.
graphics Used only for controlling the toggle menus for the bootstrap and code styles.

Once you have decided which options you want to use, it is as simple as running the following code for your Rmd file (named file.Rmd in the example):

## Install if needed
install.packages("knitrBootstrap")

## knit with knitrBootstrap
library("knitrBootstrap")
knit_bootstrap("file.Rmd", code_style = "Brown Paper", chooser = c("boot", "code"), 
    show_code = FALSE)

Things get a tiny bit more complicated if you want to use RStudio. You basically have to modify your .Rprofile file, then load RStudio and change the settings to weave files with knitrinstead of using Sweave. Then, you have to use knitr::render_html() on the Rmd file itself. Below is a short example of the .Rprofile modified to use knitrBootstrap and the basic Rmd example.

You can view the final output here. Note that you might need to click on “hide toolbars” (a RPubs option) to clearly view the menus on the bottom.

If you are like me and use Textmate as your text editor, you can knit the Rmd files with knitrBootstrap and preview them directly on the Textmate viewr using a command like this (modified from the SWeave bundle):

Other usage options are described in the knitrBootstrap help page.

To close off, let me emphasize how useful it is to be able to generate a single report that is pleasant to the eye, contains all the information, and is easily sharable for both the R and no-R crowds. Plus it’s reproducible!

I really like this package and would like to thank Jim Hester for this great package! Keep up the good work! I even use knitrBootstrap in derfinderReport which generates reports on the results from derfinder, a package that I am currently developing.

References

Citations made with knitcitations (Boettiger, 2013).

Carl Boettiger, (2013) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations
Yihui Xie, (2013) Dynamic Documents with {R} and knitr. http://yihui.name/knitr/
Jim Hester, (2013) knitrBootstrap: Knitr Bootstrap framework.. http://CRAN.R-project.org/package=knitrBootstrap

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitrBootstrap_0.9.0 knitcitations_0.4-7  bibtex_0.3-6        
## [4] knitr_1.5           
## 
## loaded via a namespace (and not attached):
##  [1] codetools_0.2-8 digest_0.6.4    evaluate_0.5.1  formatR_0.10   
##  [5] httr_0.2        markdown_0.6.3  RCurl_1.95-4.1  stringr_0.6.2  
##  [9] tools_3.0.2     XML_3.95-0.2    xtable_1.7-1

Check other topics on #rstats.

Trying to reduce the memory overhead when using mclapply

Thu, 14 Nov 2013 00:00:00 +0000

I am currently trying to understand how to reduce the memory used by mclapply. This function is rather complicated and others have explained the differences versus parLapply (A_Skelton73, 2013; lockedoff, 2012 ) and also made it clear that in mclapply each job does not know if the others are running out of memory and thus cannot trigger gc (Urbanek, 2012).

While I still struggle to understand all the details of mclapply, I can successfully use it to reduce computation time at the expense of a very high memory load. I am still looking for tips on how to reduce this memory load.

Here is what I have done.

Problem setting

I have a large data set on the form of a data.frame. I want to apply a function that works using subsets of the data.frame without the need for communication between the chunks, and I want to apply the function fast. In other words, I can safely split the matrix and speed the computation process using mclapply.

While this works, I would like to minimize memory consumption.

Toy data

Here is just some toy data for the example.

## The real data set is much larger than this
set.seed(20131113)
data <- data.frame(matrix(rnorm(1e+05), ncol = 10))
dim(data)

## [1] 10000    10

Approach 1

The first approach I have used is to pre-split the data and then use mclapply over the split data. For illustrative purposes, lets say that the function I want to apply is just rowMeans.

## Pre-split the data
dataSplit <- split(data, rep(1:10, each = 1000))

## Approach 1
library("parallel")
res1 <- mclapply(dataSplit, rowMeans, mc.cores = 2)

This gets the job done, but because my real dataSplit is much larger in memory, using say 8-10 cores blows up the memory.

Best way to pre-split?

If I know that if I am using $ n $ number of cores (in this example $ n=2 $ ) and the data set has $ m $ rows, then one option for approach #1 is to split the data into $ n $ chunks each of size $ m / n $ (rounding if needed).

## Pre-split the data into m/n chunks
dataSplit1b <- split(data, rep(1:2, each = 5000))

## Approach 1b
res1b <- mclapply(dataSplit1b, rowMeans, mc.cores = 2)

The memory needed is then in part determined by the chunksize (1000 vs 5000 shown above). One excellent suggestion (via Ben) is to reduce the memory load using this approach is to just smaller chunks. However, the runtime of the function I want to apply (rowMeans in the example) is not very sensible to the chunksize used, thus using very small chunks is not ideal as it increases computation time. Finding the sweet point is tricky, but using chunksizes of $m / (2n) $ could certainly help memory wise without majorly affecting computation time.

Approach 2

One suggestion (via Roger) is to use an environment in order to minimize copying (and thus memory load) while using mclapply over a set of indexes.

## Save the split data in an environment
my.env <- new.env()
my.env$data1 <- dataSplit1b[[1]]
my.env$data2 <- dataSplit1b[[2]]

## Function that takes indexes, then extracts the data from the environment
applyMyFun <- function(idx, env) {
    eval(parse(text = paste0("result <- env$", ls(env)[idx])))
    rowMeans(result)
}

## Approach 2
index <- 1:2
names(index) <- 1:2
res2 <- mclapply(index, applyMyFun, env = my.env, mc.cores = 2)

## Same result?
identical(res1b, res2)

## [1] TRUE

Approach 3

Another suggestion (via Roger) is to save the data chunks and load them individually inside the function that I pass to mclapply. This does not seem ideal in terms of having to create the temporary chunk data files. But I would expect this method to have the lowest memory footprint.

## Save the chunks
for (i in names(dataSplit1b)) {
    chunk <- dataSplit1b[[i]]
    output <- paste0("chunk", i, ".Rdata")
    save(chunk, file = output)
}

## Function that loads the chunk
applyMyFun2 <- function(idx) {
    load(paste0("chunk", idx, ".Rdata"))
    rowMeans(chunk)
}

## Approach 3
res3 <- mclapply(index, applyMyFun2, mc.cores = 2)

## Same result?
identical(res1b, res3)

## [1] TRUE

Computation time comparison

Computation time wise, approaches 2 and 3 do not seem very different. Approach 1b seems a tiny bit faster. [Edit: the order of the best approach might change slightly if you re-run this code]

library("microbenchmark")
micro <- microbenchmark(mclapply(dataSplit1b, rowMeans, mc.cores = 2), mclapply(index, 
    applyMyFun, env = my.env, mc.cores = 2), mclapply(index, applyMyFun2, mc.cores = 2))
micro

## Unit: milliseconds
##                                                     expr   min    lq
##            mclapply(dataSplit1b, rowMeans, mc.cores = 2) 17.43 19.97
##  mclapply(index, applyMyFun, env = my.env, mc.cores = 2) 17.05 19.20
##               mclapply(index, applyMyFun2, mc.cores = 2) 17.19 23.11
##  median    uq   max neval
##   21.41 26.00 65.53   100
##   20.60 23.92 43.67   100
##   24.56 28.39 46.99   100

library("ggplot2")
autoplot(micro)

Memory wise comparison

Relying on the cluster tools for calculating the maximum memory used, I ran each approach (1b, 2, and 3) ten times each using 2 cores using the scripts available in this gist. The maximum memory used showed no variability (within an approach) and the results are that approach 1b used 1.224G RAM, approach 2 used 1.176G RAM, and approach 3 used 1.177G RAM. Not a huge difference. Due to having to write and then load, approach 3 was slower than the other two.

Re-doing the previous test but using 20 cores lead to very similar wall clock computation times between all three approaches and to approaches 1b and 2 for 2 cores. This is due to the nature of the example, aka rowMeans is fast even with the larger chunks. Approach 1b used 7.728G RAM, approach 2 used 7.674G RAM, and approach 3 used 7.690G RAM. Hm…

Using 20 cores with previously created data files (either the split data for approaches 1b and 2, or the chunk files for approach 3) has a very different memory footprint. Approach 1b used in average 6.0744G RAM, approach 2 used 4.2647G RAM , and approach 3 used 2.6545G RAM.

Edit

Ryan from (Ryan 2013) contributed a fourth approach which used 6.794G RAM when starting from scratch with 20 cores. This approach definitely beats the other ones under the condition of starting from scratch. Note that just creating the data object uses 558.938M RAM: multiplied by 20 it would be around 10.92G RAM.

Conclusions

Using 2 or 20 cores, approach 2 beat by a very small margin approaches 3 and 1b in terms of memory usage. However, all approaches failed in terms of not having the memory blow up as you increase the number of cores when starting from scratch.

If a lower memory option is used for splitting the data and creating the chunk files, approach 3 seems like the winner in terms of memory usage. So in pure terms of lowering the memory load on mclapply approach 3 wins, although you still need to create the chunk files and do so without much memory usage.

If you have any ideas or suggestions, please let me know! Thank you!

References

Citations made with knitcitations (Boettiger, 2013).

A_Skelton73, (2013) understanding the differences between mclapply and parLapply in R. understanding the differences between mclapply and parLapply in R - Stack Overflow http://stackoverflow.com/questions/17196261/understanding-the-differences-between-mclapply-and-parlapply-in-r
lockedoff, (2012) Using mclapply, foreach, or something else in [r] to operate on an object in parallel?. Using mclapply, foreach, or something else in [r] to operate on an object in parallel? - Stack Overflow http://stackoverflow.com/questions/11036702/using-mclapply-foreach-or-something-else-in-r-to-operate-on-an-object-in-par
[R-sig-hpc] mclapply: rm intermediate objects and returning memory . https://mailman.stat.ethz.ch/pipermail/r-sig-hpc/2012-October/001534.html
[Bioc-devel] Trying to reduce the memory overhead when using mclapply . https://stat.ethz.ch/pipermail/bioc-devel/2013-November/004930.html
Carl Boettiger, (2013) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] parallel  stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
## [1] microbenchmark_1.3-0 ggplot2_0.9.3.1      knitcitations_0.4-7 
## [4] bibtex_0.3-6         knitr_1.5           
## 
## loaded via a namespace (and not attached):
##  [1] codetools_0.2-8    colorspace_1.2-4   dichromat_2.0-0   
##  [4] digest_0.6.4       evaluate_0.5.1     formatR_0.10      
##  [7] grid_3.0.2         gtable_0.1.2       httr_0.2          
## [10] labeling_0.2       MASS_7.3-29        munsell_0.4.2     
## [13] plyr_1.8           proto_0.3-10       RColorBrewer_1.0-5
## [16] RCurl_1.95-4.1     reshape2_1.2.2     scales_0.2.3      
## [19] stringr_0.6.2      tools_3.0.2        XML_3.95-0.2      
## [22] xtable_1.7-1

Scripts

The scripts are available in this gist. The main one is testApproach.R while the other ones are just job-submitters.

Check other topics on #rstats.

Creating your Jekyll-Bootstrap powered blog for R blogging

Sat, 09 Nov 2013 00:00:00 +0000

As you might have noticed, I recently decided to move Fellgernon Bit from Tumblr to GitHub. There are a couple of reasons why I made this change.

I wanted a more professional-looking blog.
- There are not many R blogs on Tumblr, and well, long text posts are not really meant for Tumblr.
Better code highlighting.
- I had enabled R code highlighting using the highlighting instructions from Jeffrey Horner (Horner, Part I). The instructions are great! I guess I just got lazy to tweak the CSS of my Tumblr blog to fix some things I didn’t like.
- I also want to be able to highlight code from other languages, like bash.
Make it easier to embed any HTML code.
- Yes, you can write posts at Tumblr in pure HTML but sometimes things break as Tumblr doesn’t like them.
Easier process of writing posts using Rmd files.
- The steps in Horner’s guide (Horner, Part II) are quite involved. Instead of those steps, I was just knitting my posts and then manually copying the html code in the body tags, then pasting it in Tumblr.
Easy way to host figures and link to them.
- I have previously hosted pictures through Picasa Web Albums, but with the push towards Google Plus they made it harder to embed solo pictures (you can embed Google+ posts though).

I had also previously heard about Jekyll-powered blogs, specially thanks to Yihui Xie’s blog (Xie, English version). At the time when I first learned about this type of blog it seemed challenging so I never dedicated the time to really learn about it. What ended up movitivating me to do so was our new student bloggers group @jhubiostat.

Let me describe how I set up Fellgernon Bit @ GitHub.

Setting up Fellgernon Bit @ GitHub

Creating the Jekyll-Bootstrap blog

My main guide to setup the blog comes from a post by J. Fisher (Fisher, 2012) which points to another of his posts (Fisher, 2012b).

Install Jekyll

Before doing anything, I first had to install Jekyll. To do so I recommend checking the installation documentation (Jekyll Installation).

To install Jekyll, I needed to have Ruby and RubyGems. Since I use MacPorts these are the steps I used:


## Update MacPorts, can take a while!!
sudo port selfupdate
sudo port upgrade outdated

## Search for Ruby and install it
## I think that Jekyll needs 1.9+ since I had 1.8 already installed
port search ruby
sudo port install ruby19
## Set as default version
sudo port select --set ruby ruby19

## Check which version you have
ruby --version

## I manually set mine to be ruby1.9
cd /opt/local/bin
sudo ln -s ruby1.9 ruby
ruby --version 
## This is mine:
## ruby 1.9.3p448 (2013-06-27 revision 41675) [x86_64-darwin12]

With Ruby in place, I could then install jekyll.


sudo gem install jekyll

Note that you can also do this after initializing the blog.

Initialize blog

Following the instructions (Fisher, 2012b) I then acquired all the setup files for Jekyll-Bootstrap. The installation instructions are quite straight forward as you will see (JB Quick Start).

First, I created the lcolladotor.github.com repository on GitHub. As my GitHub username is lcolladotor, by default GitHub will consider the lcolladotor.github.com repository as a GitHub pages repository and publish it. Thus creating lcolladotor.github.io/.

Once the repository was created, I added the Jekyll-Bootstrap files.


## Initialize repo with Jekyll-Bootstrap files
git clone https://github.com/plusjade/jekyll-bootstrap.git lcolladotor.github.com
cd lcolladotor.github.com/
git remote set-url origin git@github.com:lcolladotor/lcolladotor.github.com.git
git push -u origin master

Change theme to Twitter-2.0

Jekyll-Bootstrap includes several themes (JB Themes) which you check on their theme explorer. I think that J. Fisher’s blog looks good with Twitter-2.0 and decided to follow his lead. Furthermore, he has customized a few things (check his blog’s history) which I just plan on using.


## Install Twitter-2.0 theme
rake theme:install git="https://github.com/gdagley/theme-twitter-2.0"

Code highlighting

The default Jekyll highlighting setup uses pygments. I am basing my statement on the documentation for the highlight argument in ?knitr::render_jekyll:

which code highlighting engine to use: for pygments, the Liquid syntax is used (default approach Jekyll)

So I installed Pygments.


## Install Pygments, assuming you have Python installed
curl -O http://python-distribute.org/distribute_setup.py
sudo python distribute_setup.py
sudo easy_install Pygments

Note that (Fisher, 2012b) details some other steps. I basically copied his syntax.css file and saved it as /assets/themes/twitter-2.0/css/syntax.css

Note that you do have to add to _includes/themes/twitter-2.0/default.html the following line:

<link href="/assets/themes/twitter-2.0/css/syntax.css" rel="stylesheet" type="text/css">

As you can see on my own file in line 23 (currently) I am using a Liquid syntax instead of specifying the full path.

Import posts

Next I had to import my posts from Tumblr. Jekyll has a varied set of tools that allow you to quickly import posts which I recommend using if you are migrating your blog (Blog Migrations).

The Tumblr importer allows you to either import them into HTML format or Markdown format. I first tried Markdown, but ran into some problems. So I decided to import them into HTML format, and rename them to Markdown.


## Install importing tools
sudo gem install jekyll-import --pre

## Import the posts
cd ~/
mkdir tmp
/opt/local/bin/ruby1.9 -rubygems -e 'require "jekyll/jekyll-import/tumblr"; JekyllImport::Tumblr.process("http://fellgernon.tumblr.com", "html", true, false, false)'
## There is a post that the converter didn't like, so I had to set to "private" mode


## Rename from HTML to Markdown
cd _posts/tumblr/
for i in *html; do echo $i; filename=$(basename "$i"); filename2="${filename%.*}"; echo $filename2; mv $i ${filename2}.md; done

## Move the posts to _posts
mv ~/tmp/_posts/tumblr/*md ~/WhereYouHaveYourLocalGitRepoCopy/lcollado.github.com/_posts/

I then had to do some manual edits on the posts, specially for code chunks. For example, this post has R chunks which I had to surround by:

To complete the posts, I made sure the YAML front matter was correct and defined categories since I only had tags in Tumblr.

Viewing your blog locally

Once you have the posts, you can use Jekyll to view your blog locally.


## Go to your directory
~/WhereYouHaveYourLocalGitRepoCopy/lcollado.github.com/

## Generate the static view of your blog
jekyll serve

In my case, jekyll serve prompted a couple of errors which I had to go to the posts and edit. For example, some HTML tag that I opened but didn’t close. Stuff like that.

Once you run the previous command, you can then view your site on your browser at http://localhost:4000/

Customizing

The very first thing you must change is the _config.yml file. The file you copied from Jekyll-Bootstrap includes instructions on how to do so. If you learn by example, take a peek at mine.

In my case, I use Disqus for comments and Google Analytics for tracking visits. Since I was also using Disqus for my Tumblr blog, I used Disqus’ tools to migrate the comments. Finally, I use FeedBurner for my RSS feed, which I was able to update to the new RSS generated by Fellgernon Bit @ GitHub. Thus, the subscribers to feeds.feedburner.com/FellgernonBit didn’t have to change anything.

I also added AddThis Smart Layers with bit.ly shortening to the site and a ClustrMap as you can see in _layouts/page.html.

As for math, I am currently using the rdiscount flavor of Markdown and loading the MathJax script as you can see in _layouts/default.html. However, note that we shoudn’t ask too much power out of Markdown for math (Xie, 2013).

To use rdiscount locally, I also had to install it. I have also tried out kramdown.


## Markdown flavors
sudo gem install rdiscount
sudo gem install kramdown

## I don't remember why I also installed this:
sudo gem install nokogiri

I then customized the index and added other pages:

index.html
- Used code from Jason Fisher to add the pagination.
about.html
- Simple description
_includes/themes/twitter-2.0/default.html
- I added a CSS drop down menu (constructed with CSS Menu Maker) with the blogs I follow. For this I also had to modify _assets/themes/twitter-2.0/css/style.css.
- I modified the footer to mention that my posts are under the Creative Commons License version 3 BY-NC-SA. An idea I took from Yihui Xie’s blog.

Since I am interested in getting my blog to be a part of R-bloggers, I created an RSS specific for the posts under the rstats category to comply with their requirements (Add your blog!).

For the category-specific RSS I used feed.category.xml (Jekyll RSS Feed Templates) and created rss-rstats.xml [see atom-rstats.xml if you prefer Atom RSS] which is then processed and converted to feeds.feedburner.com/FellgernonBit-rstats.

Note that I use FeedBurner to then create Twitter updates linking to back to the posts. Also, for the rstats one I add the #rstats hashtag to the post.

Writing an R post

J. Fisher again explains in detail how to blog using Knitr and Jekyll (Fisher, 2012). Since I use TextMate2 as my text editor, I went ahead and modified his code and created a command to do so.

Basically, I modified the knit command from the SWeave TextMate bundle and added J. Fisher’s code. I saved it under the Menu Actions of the SWeave bundle and linked it to alt + e.

For this to work, I use Google Chrome with the Markdown Preview extension installed.

This is the code for the TextMate command. You could certainly make it much more concise and drop things, but I like to see a quick preview of the post. However, I still recommend using jekyll serve for a final thorough preview of the post before pushing it to GitHub.

Things I might add

Carl Boettiger has a great site –specially the Lab Notebook part– and he describes all the tools he used for making it (Boettiger, 2013). One of the features I would like to add to my site is the history button on the right sidebar he has on his posts which link to the GitHub history of the post. For example check this post and its history. I think that it can be very useful for those interested in any edits made to the posts. For example, I had to fix a couple of things on one of my recent posts as you can see here.

Looking around, I found the code needed for this to work in Carl’s repository: _includes/sidebar.html, _plugins/jekyll-labnotebook-plugins.

However, GitHub currently does not support Jekyll plugins. Meaning that if you use any plugins, you have to create the site locally and then push it to GitHub instead of just pushing the new posts and letting GitHub handle creating the site. It’s not a huge deal, but for now I’ll rely on GitHub and keep things simple for me.

If I were to do it again

Once you install Jekyll, I think the easiest way to set up your own blog is to fork another person’s blog

J. Fisher’s,
Yihui Xie’s,
Carl Boettiger’s,
mine,
or someone else blog you like.

Then delete their posts, give attribution, and customize things like the blog title and other things you would have to customize on any blogging platform (RSS, comments provider, etc).

Obviously Jekyll gives you much more control over your site, but at the same time, you must also dedicate enough time to learn what to change and make sure that things don’t break.

Post template

To finish describing my setup, here is my post template.

Wrap up

I hope that you will find some of this information useful if you are thinking of starting your R blog powered by Jekyll-Bootstrap and hosted by GitHub.

Note that since the blog is hosted in GitHub, others can send you pull requests to fix things and/or create guest posts. Just as easily you can make it a multi-author blog by changing the collaborators setting of the GitHub repository.

This wouldn’t have been possible without J. Fisher’s posts and many other resources available online.

References

Citations made with knitcitations (Boettiger, 2013b).

@jeffreyhorner, Blog with R Markdown and tumblr: Part I. Jeffrey Horner http://jeffreyhorner.tumblr.com/post/25804518110/blog-with-r-markdown-and-tumblr-part-i
@jeffreyhorner, Blog with R Markdown and tumblr: Part II. Jeffrey Horner http://jeffreyhorner.tumblr.com/post/25943954723/blog-with-r-markdown-and-tumblr-part-ii
Yihui Xie, Statistics, R, Graphics and Fun | Yihui Xie. http://yihui.name/en/
Jason Fisher, (2012) Blog with Knitr and Jekyll. http://jfisher-usgs.github.io/r/2012/07/03/knitr-jekyll/
Jason Fisher, (2012) Jekyll Build on Windows. http://jfisher-usgs.github.io/lessons/2012/05/30/jekyll-build-on-windows/
Installation. Jekyll â€¢ Simple, blog-aware, static sites http://jekyllrb.com/docs/installation/
Jekyll Quick Start | ruhoh universal static blog generator. http://jekyllbootstrap.com/usage/jekyll-quick-start.html
Using Themes | ruhoh universal static blog generator. http://jekyllbootstrap.com/usage/jekyll-theming.html
Blog migrations. Jekyll â€¢ Simple, blog-aware, static sites http://jekyllrb.com/docs/migrations/
Tal Galili, add your blog!. R-bloggers http://www.r-bloggers.com/add-your-blog/
jekyll-rss-feeds. GitHub https://github.com/snaptortoise/jekyll-rss-feeds
Yihui Xie, (2013) Markdown or LaTeX? | Yihui Xie. http://yihui.name/en/2013/10/markdown-or-latex/
Carl Boettiger, (2013) About this site. Lab Notebook http://carlboettiger.info/README.html
Carl Boettiger, (2013) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitcitations_0.4-7 bibtex_0.3-6        knitr_1.5          
## 
## loaded via a namespace (and not attached):
##  [1] codetools_0.2-8 digest_0.6.4    evaluate_0.5.1  formatR_0.10   
##  [5] httr_0.2        RCurl_1.95-4.1  stringr_0.6.2   tools_3.0.2    
##  [9] XML_3.95-0.2    xtable_1.7-1

ggplot Tutorial

Fri, 21 Jun 2013 00:00:00 +0000

ggplot Tutorial

I liked the following ggplot2 tutorial which is featured in Gabriela de Queiroz’s blog called unbiasedestimator. The tutorial looks very neatly presented and I’m sure that it will be very helpful to anyone just getting started with ggplot2 before they jump into ggplot2: Elegant Graphics for Data Analysis by Hadley Wickham or R Graphics Cookbook by Winston Chang.

The tutorial is very nicely formatted with code in bold highlighting parts that change something in the plot. Overall, the tutorial explains how to use qplot() although it does have a longer example using ggplot() to make survival curves.

Check it out!

unbiasedestimator:

Good tutorial about the R package.

“ggplot2 is an R package for producing statistical, or data, graphics, but it is unlike most other graphics packages because it has a deep underlying grammar. [..]”

- H.Wickham, ggplot2, Use R, DOI 10.1007/978-0-387-98141_1, © Springer Science+Business Media, LLC 2009 -

userR2013 data analysis contest: data exploration

Wed, 12 Jun 2013 00:00:00 +0000

Description

The useR2013 conference is organizing a data analysis contest, check the rules here.

They have a package called useR2013DAC with two data sets: one from La Liga and the other one from the Formula 1. Once you download and install the package (available here), you can quickly explore the data using the following R commands:

Data exploration

## Load the package
library(useR2013DAC)

## Explore laliga data
data(laliga)
head(laliga)

##    Season Week               HomeTeam                    AwayTeam
## 1 2008/09    1   Athletic Club Bilbao     Union Deportiva Almeria
## 2 2008/09    1        Atlético Madrid                   Málaga CF
## 3 2008/09    1          Betis Sevilla Real Club Recreativo Huelva
## 4 2008/09    1             CA Osasuna               Villarreal CF
## 5 2008/09    1            CD Numancia                FC Barcelona
## 6 2008/09    1 Deportivo de La Coruña              Real Madrid CF
##   HomeGoals AwayGoals
## 1         1         3
## 2         4         0
## 3         0         1
## 4         1         1
## 5         1         0
## 6         2         1

summary(laliga)

##     Season               Week        HomeTeam           AwayTeam        
##  Length:1900        Min.   : 1.0   Length:1900        Length:1900       
##  Class :character   1st Qu.:10.0   Class :character   Class :character  
##  Mode  :character   Median :19.5   Mode  :character   Mode  :character  
##                     Mean   :19.5                                        
##                     3rd Qu.:29.0                                        
##                     Max.   :38.0                                        
##                                                                         
##    HomeGoals      AwayGoals   
##  Min.   :0.00   Min.   :0.00  
##  1st Qu.:1.00   1st Qu.:0.00  
##  Median :1.00   Median :1.00  
##  Mean   :1.65   Mean   :1.14  
##  3rd Qu.:2.00   3rd Qu.:2.00  
##  Max.   :8.00   Max.   :8.00  
##  NA's   :50     NA's   :50

lapply(laliga, class)

## $Season
## [1] "character"
## 
## $Week
## [1] "integer"
## 
## $HomeTeam
## [1] "character"
## 
## $AwayTeam
## [1] "character"
## 
## $HomeGoals
## [1] "integer"
## 
## $AwayGoals
## [1] "integer"

## Explore formula1 data
data(formula1)
head(formula1)

##   Pos No             Driver             Team Laps        Time Grid Pts
## 1   1  8    Fernando Alonso          Ferrari   49 1:39:20.396    3  25
## 2   2  7       Felipe Massa          Ferrari   49  +16.0 secs    2  18
## 3   3  2     Lewis Hamilton McLaren-Mercedes   49  +23.1 secs    4  15
## 4   4  5   Sebastian Vettel      RBR-Renault   49  +38.7 secs    1  12
## 5   5  4       Nico Rosberg      Mercedes GP   49  +40.2 secs    5  10
## 6   6  3 Michael Schumacher      Mercedes GP   49  +44.1 secs    7   8
##                                         Race Season
## 1 2010 FORMULA 1 GULF AIR BAHRAIN GRAND PRIX   2010
## 2 2010 FORMULA 1 GULF AIR BAHRAIN GRAND PRIX   2010
## 3 2010 FORMULA 1 GULF AIR BAHRAIN GRAND PRIX   2010
## 4 2010 FORMULA 1 GULF AIR BAHRAIN GRAND PRIX   2010
## 5 2010 FORMULA 1 GULF AIR BAHRAIN GRAND PRIX   2010
## 6 2010 FORMULA 1 GULF AIR BAHRAIN GRAND PRIX   2010

summary(formula1)

##       Pos            No                     Driver    
##  Ret    :254   1      :  58   Felipe Massa     :  58  
##  1      : 58   10     :  58   Fernando Alonso  :  58  
##  10     : 58   11     :  58   Heikki Kovalainen:  58  
##  11     : 58   12     :  58   Jenson Button    :  58  
##  12     : 58   14     :  58   Kamui Kobayashi  :  58  
##  13     : 58   15     :  58   Lewis Hamilton   :  58  
##  (Other):848   (Other):1044   (Other)          :1044  
##                    Team          Laps             Time          Grid     
##  Ferrari             :116   55     :125   +1 Lap    :268   1      :  58  
##  Force India-Mercedes:116   56     :121   +2 Laps   :102   10     :  58  
##  HRT-Cosworth        :116   53     : 92   Accident  : 93   11     :  58  
##  McLaren-Mercedes    :116   57     : 80   +3 Laps   : 41   12     :  58  
##  STR-Ferrari         :116   70     : 75   Hydraulics: 26   13     :  58  
##  Lotus-Renault       : 78   52     : 69   Gearbox   : 24   14     :  58  
##  (Other)             :734   (Other):830   (Other)   :838   (Other):1044  
##       Pts          Race               Season    
##         :812   Length:1392        Min.   :2010  
##  1      : 58   Class :character   1st Qu.:2010  
##  10     : 58   Mode  :character   Median :2011  
##  12     : 58                      Mean   :2011  
##  15     : 58                      3rd Qu.:2012  
##  18     : 58                      Max.   :2012  
##  (Other):290

lapply(formula1, class)

## $Pos
## [1] "factor"
## 
## $No
## [1] "factor"
## 
## $Driver
## [1] "factor"
## 
## $Team
## [1] "factor"
## 
## $Laps
## [1] "factor"
## 
## $Time
## [1] "factor"
## 
## $Grid
## [1] "factor"
## 
## $Pts
## [1] "factor"
## 
## $Race
## [1] "character"
## 
## $Season
## [1] "numeric"

I don’t see a specific question that they want you to answer with this data, but if you find one related to data analysis or visualization then join the competition!

Note that you must be attending the conference in order to be eligible to compete.

Reproducibility

```r sessionInfo() ``` ```r ## R version 3.0.0 (2013-04-03) ## Platform: x86_64-apple-darwin10.8.0 (64-bit) ## ## locale: ## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 ## ## attached base packages: ## [1] stats graphics grDevices utils datasets methods base ## ## other attached packages: ## [1] useR2013DAC_0.1-1 knitr_1.2 ## ## loaded via a namespace (and not attached): ## [1] digest_0.6.3 evaluate_0.4.3 formatR_0.7 stringr_0.6.2 ## [5] tools_3.0.0 ```

Reading an R file from GitHub

Thu, 09 May 2013 00:00:00 +0000

Lets say that I want to read in this R file from GitHub into R.

The first thing you have to do is locate the raw file. You can do so by clicking on the Raw button in GitHub. In this case it’s https://raw.github.com/lcolladotor/ballgownR-devel/master/ballgownR/R/infoGene.R

One would think that using source() would work, but it doesn’t as shown below:

```r source("https://raw.github.com/lcolladotor/ballgownR-devel/master/ballgownR/R/infoGene.R") ``` ```r ## Warning: unsupported URL scheme ``` ```r ## Error: cannot open the connection ```

However, thanks again to Hadley Wickham you can do so by using the devtools (Wickham & Chang, 2013 ) package.

Here is how it works:

```r library(devtools) library(roxygen2) ## Needed because this file has roxygen2 comments. Otherwise you get a ## 'could not find function 'digest'' error source_url("https://raw.github.com/lcolladotor/ballgownR-devel/master/ballgownR/R/infoGene.R") ``` ```r ## SHA-1 hash of file is 6c32a620799eded5d6ff0997a184843d7964724a ``` ```r ## Note that you can specify the SHA-1 hash to be very specific about ## which version of the file you want to read in. ```

We can then check that infoGene has actually been sourced:

```r "infoGene" %in% ls() ``` ```r ## [1] TRUE ```

That’s it! Enjoy!

Citations made with knitcitations (Boettiger, 2013 ).

Hadley Wickham, Winston Chang, (2013) devtools: Tools to make developing R code easier. http://CRAN.R-project.org/package=devtools
Carl Boettiger, (2013) knitcitations: Citations for knitr markdown files. https://github.com/cboettig/knitcitations

Reproducibility

```r sessionInfo() ``` ```r ## R version 3.0.0 (2013-04-03) ## Platform: x86_64-apple-darwin10.8.0 (64-bit) ## ## locale: ## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 ## ## attached base packages: ## [1] stats graphics grDevices utils datasets methods base ## ## other attached packages: ## [1] roxygen2_2.2.2 digest_0.6.3 devtools_1.2 ## [4] knitcitations_0.4-6 bibtex_0.3-5 knitr_1.2 ## ## loaded via a namespace (and not attached): ## [1] brew_1.0-6 evaluate_0.4.3 formatR_0.7 httr_0.2 ## [5] memoise_0.1 parallel_3.0.0 RCurl_1.95-4.1 stringr_0.6.2 ## [9] tools_3.0.0 whisker_0.3-2 XML_3.95-0.2 xtable_1.7-1 ```

Using plyr and doMC for quick and easy apply-family functions

Fri, 26 Apr 2013 00:00:00 +0000

A few weeks back I dedicated a short amount of time to actually read what plyr (Wickham, 2011) is about and I was surprised. The whole idea behind plyr is very simple: expand the apply() family to do things easy. plyr has many functions whose name ends with ply which is short of apply. Then, the functions are identified by two letters before ply which are abbreviations for the input (first letter) and output (second one). For instance, ddply takes an input a data.frame and returns a data.frame while ldply takes as input a list and returns a data.frame.

The syntax is pretty straight forward. For example, here are the arguments for ddply:

library(plyr)
args(ddply)
## function (.data, .variables, .fun = NULL, ..., .progress = "none", 
##     .inform = FALSE, .drop = TRUE, .parallel = FALSE, .paropts = NULL) 
## NULL

What we basically have to specify are

.data which in general is the name of the input data.frame,
.variables which is a vector (note the use of the . function) of variable names. In this case, ddply is very useful for applying some function to subsets of the data as specified by these variables,
.fun which is the actual function we want to run,
and ... which are parameter options for the function we are running.

From the ddply help page we have the following examples:

dfx <- data.frame(
  group = c(rep('A', 8), rep('B', 15), rep('C', 6)),
  sex = sample(c("M", "F"), size = 29, replace = TRUE),
  age = runif(n = 29, min = 18, max = 54)
)

# Note the use of the '.' function to allow
# group and sex to be used without quoting
ddply(dfx, .(group, sex), summarize,
 mean = round(mean(age), 2),
 sd = round(sd(age), 2))
##   group sex  mean    sd
## 1     A   F 40.48 12.72
## 2     A   M 34.48 15.28
## 3     B   F 36.05  9.98
## 4     B   M 38.35  7.97
## 5     C   F 20.04  1.86
## 6     C   M 43.81 10.72

# An example using a formula for .variables
ddply(baseball[1:100, ], ~year, nrow)

##   year V1
## 1 1871  7
## 2 1872 13
## 3 1873 13
## 4 1874 15
## 5 1875 17
## 6 1876 15
## 7 1877 17
## 8 1878  3

# Applying two functions; nrow and ncol
ddply(baseball, .(lg), c("nrow", "ncol"))

##   lg  nrow ncol
## 1       65   22
## 2 AA   171   22
## 3 AL 10007   22
## 4 FL    37   22
## 5 NL 11378   22
## 6 PL    32   22
## 7 UA     9   22

But this is not the end of the story! Something I really liked about plyr is that it can be parallelized via the foreach (Analytics, 2012) package. I don’t know much about foreach, but all I learnt is that you have to use other packages such as doMC (Analytics, 2013) to actually run the code. It’s like foreach specifies the infraestructure to communicate in parallel (and split jobs) and packages like doMC tailor it for specific environments like for running in multi-core.

Running things in parallel can then be very easy. Basically, you load the packages, specify the number of cores, and run your ply function. Here is a short example:

## Load packages
library(plyr)
library(doMC)

## Loading required package: foreach
## Loading required package: iterators
## Loading required package: parallel

## Specify the number of cores
registerDoMC(4)

## Check how many cores we are using
getDoParWorkers()
## [1] 4

## Run your ply function
ddply(dfx, .(group, sex), summarize, mean = round(mean(age), 2), sd = round(sd(age), 
    2), .parallel = TRUE)

##   group sex  mean    sd
## 1     A   F 40.48 12.72
## 2     A   M 34.48 15.28
## 3     B   F 36.05  9.98
## 4     B   M 38.35  7.97
## 5     C   F 20.04  1.86
## 6     C   M 43.81 10.72

In case that you are interested, here is a short shell script for knitting an Rmd file in the cluster and specifying the appropriate number of cores to then use plyr and doMC.

#!/bin/bash 
# To run it in the current working directory
#$ -cwd 
# To get an email after the job is done
#$ -m e 
# To speficy that we want 4 cores
#$ -pe local 4
# The name of the job
#$ -N myPlyJob

echo "**** Job starts ****"
date

# Knit your file: assuming it's called FileToKnit.Rmd
Rscript -e "library(knitr); knit2html('FileToKnit.Rmd')"

echo "**** Job ends ****"
date

Lets say that the bash script is named script.sh. Then you can submit it to the cluster queue using


qsub script.sh

This is what I used to re-format a large data.frame in a few minutes in the cluster for the #jhsph753 class homework project.

So, thank you again Hadley Wickham for making awesome R packages!

Citations made with knitcitations (Boettiger, 2013).

Revolution Analytics, (2013) doMC: Foreach parallel adaptor for the multicore package. http://CRAN.R-project.org/package=doMC
Revolution Analytics, (2012) foreach: Foreach looping construct for R. http://CRAN.R-project.org/package=foreach
Carl Boettiger, knitcitations: Citations for knitr markdown files. https://github.com/cboettig/knitcitations
Hadley Wickham, (2011) The Split-Apply-Combine Strategy for Data Analysis. Journal of Statistical Software 40 (1) http://www.jstatsoft.org/v40/i01/

Predicting who will win a NFL match at half time

Sat, 23 Mar 2013 00:00:00 +0000

It was great to have a little break, Spring break, although the weather didn’t feel like spring at all! During the early part of the break I worked on my final project for Jeff Leek’s data analysis class, which we call 140.753 here. Continuing my previous posts on the topic, this time I’ll share the results of my final project.

At the beginning of the course, we had to submit a project plan (more like a proposal) and in mine I announced my interest to look into some sports data. At the time I included a few links to Brian Burke’s Advanced NFL Stats site (Burke). At the time I didn’t know that Burke’s site described in detail a lot of the information I would end up using.

My final project had to do with splitting NFL games by half and then use only the play-by-play data from the first half to predict if team A or B would win the game. My overall goal was to have some fun with sports data which I had never looked at, but then also try to come up with something I would personally use in the future. So, why split games by half? I personally would like to know if I should keep watching a game or not at half time. Having a tool to help me decide would be great, and well, if the team I’m rooting for has high chances of losing or winning, ideally I would switch to doing something else. A related question that I didn’t try to answer is which half is worth watching? This would be a meaningful question if you only have time to watch one of them.

To truly satisfy my goals, it wasn’t enough to just build a predictive model. That is why I also built a web application using the shiny package (RStudio and Inc., 2013). It was the first time I did a shiny app, but thanks to the good manual and some examples on GitHub from John Muschelli like his Shiny_model it wasn’t so bad. I thus invite you to test and browse my shiny app at http://glimmer.rstudio.com/lcolladotor/NFLhalf/. It could be improved by adding some functions that scrape live data for the 2013 season so you don’t have to input all the variables needed by using the sliders. Anyhow, I’m happy with the result.

The entire project’s code, EDA steps, shiny app, and report are available via GitHub in my repository (lcollado753). While the details are in the report, I’ll give a brief summary here.

Basically, I summarized the play-by-play data for all NFL games from 2002 to 2012 seasons as provided by Burke (Burke, 2010). I used some of the variables Burke uses (Burke, 2009) and some others like the score difference, who starts the second half, and the game day winning percentages of both teams. After exploring the data, I discarded the years 2002 to 2005. Then, I trained a model using the 2006 to 2011 data and did some quick model selection. Note that I’m not doing the adjustment by opponent the way Burke did it (Burke, 2009-2) in part because I was running out of time, but also because the model already uses the current game winning percentages of both teams to consider the two team’s strength. I evaluated the model using the 2012 data and after seeing that it worked decently enough, I trained a second model using the data from 2006 to 2012 so it can be used for the 2013 season. These two trained models are the ones available in the shiny app I made.

In the report, I didn’t include ROCs—a big miss—so here they go. The code I will show below is heavily based on a post on GLMs (denishaine, 2013). The code below is written in a way that you can easily reproduce it if you have cloned my repository for the 140.753 class (lcollado753).

First, some setup steps.

## Specify the directory where you cloned the lcollado753 repo
maindir <- "whereYouClonedTheRepo"
## Load packages needed
suppressMessages(library(ROCR))
library(ggplot2)

## Load fits.
## Remember that 1st one used data from 2006 to 2011
## and the 2nd one used data from 2006 to 2012.
load(paste0(maindir, "/lcollado753/final/nfl_half/EDA/model/fits.Rdata"))

Next, I make the ROCs for both trained models using the data that they were trained on. They should be quite good since it uses the same data to build the model that it will then try to predict.

## Make the ROC plots

## Simple list where I'll store all the results so I can compare the ROC plots later on
all <- list()

## Construct prediction function
for(i in 1:2) {
	## Predict on the original data
	pred <- predict(fits[[i]])
	
	## Subset original data (remove NA's)
	data <- fits[[i]]$data
	data <- data[complete.cases(data),]
	
	## Construct prediction function
	pred.fn <- prediction(pred, data$win)
	
	## Get performance info
	perform <- performance(pred.fn, "tpr", "fpr")
	
	## Get ready to plot
	toPlot <- data.frame(tpr = unlist(slot(perform, "y.values")), fpr = unlist(slot(perform, "x.values")))
	all <- c(all, list(toPlot))

	## Make the plot
	res <- ggplot(toPlot) + geom_line(aes(x=fpr, y=tpr)) + geom_abline(intercept=0, slope=1, colour="orange") + ylab("Sensitivity") + xlab("1 - Specificity") + ggtitle(paste("Years 2006 to", c("2011", "2012")[i]))
	print(res)
	
	## Print the AUC value
	print(unlist(performance(pred.fn, "auc")@y.values))
}

```r ## [1] 0.8506 ```

## [1] 0.8513

Both ROC plots look pretty similar (well, the data sets are very similar!) and have relatively high AUC values.

Next, I make the ROC plot using the model trained with the data from 2006 to 2011 to predict the outcomes for the 2012 games.

## Load 2012 data
load(paste0(maindir, "/lcollado753/final/nfl_half/data/pred/info2012.Rdata"))

## Predict using model fit with data from 2006 to 2011
pred <- predict(fits[[1]], info2012)

## Construction prediction function
pred.fn <- prediction(pred, info2012$win)

## Get performance info
perform <- performance(pred.fn, "tpr", "fpr")

## Get ready to plot
toPlot <- data.frame(tpr = unlist(slot(perform, "y.values")), fpr = unlist(slot(perform, "x.values")))
all <- c(all, list(toPlot))

## Make the plot
ggplot(toPlot) + geom_line(aes(x=fpr, y=tpr)) + geom_abline(intercept=0, slope=1, colour="orange") + ylab("Sensitivity") + xlab("1 - Specificity") + ggtitle("Model trained 2006-2011 predicting 2012")

## Print the AUC value
print(unlist(performance(pred.fn, "auc")@y.values))
## [1] 0.816

The steps in the curve are more visible since it is using less data. It also seems to be a little less good than the other two, as expected. This is clear when comparing the AUC values.

Finally, I plot all curves in the same picture to visually compare them.

names(all) <- c("train2011", "train2012", "pred2012")
for(i in 1:3) {
	all[[i]] <- cbind(all[[i]], rep(names(all)[i], nrow(all[[i]])))
	colnames(all[[i]])[3] <- "set"
}
all <- do.call(rbind, all)

ggplot(all) + geom_line(aes(x=fpr, y=tpr, colour=set)) + geom_abline(intercept=0, slope=1, colour="orange") + ylab("Sensitivity") + xlab("1 - Specificity") + ggtitle("Comparing ROCs")

Both ROCs with the trained data (train2011, train2012) are nearly identical and both are slightly superior to the one predicting the 2012 games.

Overall I am happy with the results and while some things can certainly be improved, I look forward to the NFL 2013 season. Also, remember that Burke publishes his winning estimated probabilities from week 4 onward (The Fifth Down Blog). So you might be interested on comparing the probability at half time versus his estimated probability which is calculated before the game starts. I mean, maybe you could use the difference between the two to have an idea of how unexpected the first half was. After all, if a game falls outside the pattern it might be worth watching.

Citations made with knitcitations (Boettiger, 2013).

lcolladotor, lcollado753. GitHub https://github.com/lcolladotor/lcollado753/tree/master/final/nfl_half
denishaine, (2013) Veterinary Epidemiologic Research: GLM – Evaluating Logistic Regression Models (part 3). denis haine http://denishaine.wordpress.com/2013/03/19/veterinary-epidemiologic-research-glm-evaluating-logistic-regression-models-part-3/
Advanced NFL Stats. http://www.advancednflstats.com/
(2010) Advanced NFL Stats: Play-by-Play Data. http://www.advancednflstats.com/2010/04/play-by-play-data.html
(2009) Advanced NFL Stats: How the Model Works–A Detailed Example Part 1. http://www.advancednflstats.com/2009/01/how-model-works-detailed-example.html
(2009) Advanced NFL Stats: How the Model Works–A Detailed Example Part 2. http://www.advancednflstats.com/2009/01/how-model-works-detailed-example-part-2.html
By BURKE, Brian Burke - The Fifth Down Blog - NYTimes.com. The Fifth Down Â» Brian Burke http://fifthdown.blogs.nytimes.com/author/brian-burke/
Carl Boettiger, knitcitations: Citations for knitr markdown files. https://github.com/cboettig/knitcitations
RStudio , Inc. , (2013) shiny: Web Application Framework for R. http://CRAN.R-project.org/package=shiny

And so begins English Composition I

Thu, 21 Mar 2013 00:00:00 +0000

This week started the English Composition I: Achieving Expertise course (Comer, 2013) that I have been looking forward to.

I am not sure yet how long I will last, but I hope to enjoy it as much as I can. Plus, it should help me with my posting and other writing areas. While I last in the course, I plan to publish my writings in the blog too. So you will hopefully see me be more active here.

As it is important to cite when writing, I have also figured out how to do so automatically in Rmd files. For that I learnt how to use knitcitations from the GitHub instructions (knitcitations) and a explanatory post (Boettiger, 2013).

knitcitations is great, but it kind of struggles with some pages. That is why I modified my template in FBit by writing my own citing function for pages where citep fails. Here is the code:

## I made my own citing function since citep() doesn't work like I want to with
## urls that are not really pages themselve like part of a GitHub repo.
mycitep <- function(x, short, year=substr(date(), 21, 24), tooltip=TRUE) {
	tmp <- citep(x)
	res <- gsub("></a>", paste0(">", short, "</a>"), tmp)
	if(tooltip) {
		res <- gsub("\\?\\?\\?\\?", year, res)
	}
	res
}

## You already saw an inline working example in the post itself.

Carl Boettiger, (2013) knitcitations. Lab Notebook http://www.carlboettiger.info/2012/05/30/knitcitations.html
cboettig, knitcitations. GitHub https://github.com/cboettig/knitcitations
Coursera. Coursera https://www.coursera.org/course/composition

Analyzing SimplyStatistics visits info

Sat, 09 Mar 2013 00:00:00 +0000

Recently we had to analyze the data of the number of visits per day to SimplyStatistics.org. There were two goals:

Estimate the fraction of visitors retained after a spike in the number of visitors
Identify (if any) any factors that influence the fraction estimated in 1.

For me it was a fun project in part because I like SimplyStatistics but also because I think that finding the answers to the questions would be interesting and help understand the readers of that blog.

Sadly, I didn’t work on it much. We had lots of stuff due that week, but well, I’m happy enough with the analysis I did. My own report is hosted here and this is the pdf file of the report itself.

Half joking with other students, I said that I basically did t-tests. Hopefully I can work on changing this tendency with the pile of recommended books I’ve been acquiring but not really reading through. Except for the ggplot2: Elegant Graphics for Data Analysis and the R Graphics Cookbook. Sounds like spring break will be fun :P

Kind of related to this, Jeff Leek announced yesterday that he is going to compile a list of student blogs that have something to do with statistics and data. He added a link to my blog which is why I saw a large peak of Fellgernon Bit’s visitor data. After all, when doing the data analysis described above I played around with the data from Fellgernon Bit and now know that at a minimum posting drives visitor’s into sites (which sounds obvious, but maybe you get random traffic) —see fig 1 of the report.

Had Jeff done so before, I could have a point estimate (but without being able to say something about the uncertainty of it) that SimplyStatistics has 142 visitors that read the posts AND click on the links. Maybe using the info from Hilary’s and Alyssa’s blogs we could have an estimate with some measure of uncertainty, but only for March 8th.

Sharing my work for "Advanced Methods III"

Wed, 13 Feb 2013 00:00:00 +0000

This semester I’m taking the live version of the Data Analysis class by Jeff Leek. His more popular version of the course is available through Coursera.

One of the things that Jeff promotes is reproducibility and sharing code. I share that tendency and thus created a Git repository for my homework and code for the class: lcollado753. I’m hosting it with GitHub to try it out since I started with Mercurial via Bitbucket.

Part of me would love it if everyone in the class had their own Git repositories. I mean, this class involves lots of practice exercises and there are plenty of R packages and functions that others use that I would like to learn. As I don’t see this happening, I think that it would be great to list the packages/functions you think could be interesting to others at the end of the write-ups. However, this involves sharing the reports and I don’t know if that will happen.

But maybe I didn’t get the instructions Jeff gave correctly the first time. Listening into his week 2 talks from the Coursera course, I get that he wants our reports to be reproducible. The idea is great, but sometimes I get lots in the technicalities of finding the best fit for our situation. Aka, something we can all do that is worth the time for small scale projects that we have a couple of days to complete and most likely will be finishing the day before they are due. For now we might stick to sharing zip files with the report + summarized data set (it has be small enough to be sharable by email).

I’m pretty happy with hosting my stuff at GitHub. One blunder I made in the first data analysis report is that I completely forgot to say in it that I have the code in GitHub :P Oh well, next time!

I feel that I also have lots to improve regarding how to tell a story in a report. Plus, for this first project I mainly did some exploratory data analysis without much stat analysis.

Overall, I’m quite excited with this course =) and I think that I’ll learn a ton on methods to analyze data AND how to actually implement them. Plus, I’m currently trying to learn ggplot2 as you can see in that first report. Also, I made it with knitr instead of Sweave =)

Introduction to R and Biostatistics (2012 version): presentation

Mon, 12 Nov 2012 00:00:00 +0000

To follow my Introducing R and Biostatistics to first year LCG students (2012 version) post, you can now find the presentation online from my site either in presentation format, in a single webpage format, or the raw Rmd file. To prove the point that publishing to RPubs is super easy, you can also find the single webpage format over there. I also like how you can comment and share in RPubs.

One of the challenges of giving a presentation to first year students is finding the balance between introducing them to cool things you are doing in your work and actually giving a talk that they can follow. I thought about this and ended dropping anything related to my work.

My presentation was split pretty much in two parts. First, I wanted to promote some philosophical discussion about what is statistics. Second, I gave a brief overview of what you can do with R. Or more exactly, what they should be able to learn to do even if they become wet biologists.

While planning this presentation, I knew that I wanted to give the new students a flavor of the three different currents in statistics. I aimed to improve my 2011 explanations now that I’m taking the Foundations of Statistical Inference course. I’m happy with the result and I think this is greatly due to Royall’s diagnostic test example.

Another key point that I wanted to emphasize was that RStudio is the way to go if you are new to R. It is very straightforward to use, plus it is nicely interegrated with knitr.

I decided to use R Markdown (Rmd) for the first time, after seeing how easy Markdown really is compared to using LaTeX and Beamer. However, when it got to doing the presentation I have to say that I was a bit dissapointed by how some things just break when using the R Markdown to Markdown to HTML presentation pipeline —using pandoc for the last step. For example, the math breaks when using mathml or mathjax at times (like after adding an iframe for a youtube video), so I had to use webtex which doesn’t look as nice.

If you are interested in the commands, I used the “Knit HTML” button in R Studio [equivalent to running from R: library(knitr); knit(“filename.Rmd”)] and then ran the following command:

pandoc -s -S —webtex -i -t dzslides intro_R_Biostat_LCG_2012.md -o intro_R_Biostat_LCG_2012_slides.html & open intro_R_Biostat_LCG_2012_slides.html

I was originally aiming to have a single Rmd file to produce an HTML presentation and a Beamer presentation. However, controlling the pictures in the Beamer output proved to be challenging. While I had a work around, the final problem was the math part. By the time I realized this it was too late —I just dropped the Beamer presentation. Googling, even the author of knitr acknowledges that the best input for PDF output is still LaTeX.

In the end, I’m happy that I got the HTML presentation done using R Markdown and briefly introduced it to the first year students. The basics of knitr are very easy to learn and I’m hoping that it got some of them curious enough to try it.

Next thing in line: prepare 5 questions for the students.

Introducing R and Biostatistics to first year LCG students (2012 version)

Tue, 30 Oct 2012 00:00:00 +0000

On Friday November 9th I’ll be giving a talk to the first year students from the Undergraduate Program on Genomic Sciences (LCG in Spanish) during their “Seminar 1: Introduction to Bioinformatics” course. It’s just like I did a year ago as I documented in my post Introducing Biostatistics to first year LCG students.

Well, this time I’ll change things a bit. I’m allowed to require the students to read 2-3 papers before my talk to introduce them to my field. I’ll do so, but in a more peculiar way by requiring them to listen in to a few videos I selected. So, without further ado here are the three required “papers”:

Here is “paper 1" (~30 minutes). The goal is to introduce you to the basic workings of R and also to great sources of R videos.

First, learn to install R (watch it in full screen).

Or you can also watch any of the two following videos:

Next learn about RStudio and why it’s a great place to start (watch it on hd and fullscreen).

Now you are ready to learn how to create a variable in R. Use RStudio instead of the R GUI to do so.

Next, learn the super basics about the basic R plot system.

Now you are ready to learn about how to use the combine function.

Next, learn about data.frame type of objects

and how to add new variables to them.

Next up is learning how to find help.

Almost there. Now check how to change your current working directory.

Finally, learn how to install and load a package in R.

If you are more curious regarding the origins of R check the next video (not part of “paper 1”).

Next, “paper 2" (~39 minutes). The goal here is to get a feeling of how you can use R to create plots.

First start with this demonstration of the basic R plotting tools (called “base graphics”). It does in enough level of detail of how the basic plotting system works and how you can customize the colors, layout, etc. For the purpose of getting used to the tool, I recommend that you follow this video using RStudio. Also, you’ll want to watch it in 720p.

Now check the demo for plotting with the lattice package. This is more advanced, but it should also be more illustrative of the power you have with R. Plus it shows how we can expand the functionality of R by using packages contributed to the community and freely available for us to use.

Finally, “paper 3" (~28 minutes). This is the first lecture from a course by Brian Caffo in which he goes over the definition and overall motivation behind Biostatistics. It should be much more fun to watch than reading a review paper in the area.

Now, for those motivated to learn more, I recommend some of my own posts summarizing information that can be useful to you.

Sources:

http://www.twotorials.com/ by Anthony Damico
Youtube videos by Roger Peng
Youtube videos by Brian Caffo

Visualizing colors()

Fri, 19 Oct 2012 00:00:00 +0000

The other day I learnt about the existance of the colors() vector in R which specifies all the character-based colors like “light blue”, “black”, etc. So I made a simple plot to visualize them all. Here’s the code:

mat <- matrix(1:length(colors()), ncol = 9, byrow= TRUE)
df <- data.frame(col = colors(), 
	x = as.integer(cut(1:length(colors()), 9)),
	y = rep(1:73, 9), stringsAsFactors=FALSE)
plot(y ~ jitter(x), data = df, col = df$col,
 	pch=16, main = "Visualizing colors() split in 9 groups",
 	xlab = "Group", 
	ylab = "Element of the group (min = 1, max = 73)",
	sub = "x = 3, y = 1 means that it's the 2 * 73 + 1 = 147th color")

And the plot:

Setting up your computer for bioinformatics/biostatistics and a compedium of resources

Thu, 23 Aug 2012 00:00:00 +0000

Jumping on the train set by Hilary Parker “The Setup (Part 1)" and Alyssa Frazee “my software/hardware setup”, I’m going to share my setup and hopefully add something new. They both did a great job already, so make sure you read their posts!

I have some experience with all three main OS: Windows, Linux and Mac. That being said, I know some of the basic stuff for each but I surely use Google very frequently to get help. I used to have a dual Windows / Linux (Ubuntu) set up but now I have a Windows laptop/desktop (it’s a monster :P) at home and I’m happy working with my Mac.

I’m going to start by mentioning the software I use(d) in each OS and then add some other tools that I really like.

Windows

Text editor: Notepad++. It outperforms Notepad by light years! A must for me is the “View -> Word wrap” option. I would definitely go to “Settings -> Preferences -> New Document/Default Directory” and change the new document format from Windows (Dos) to Unix. This will save you time later when you want to work on a Unix system like the cluster. If you didn’t, you can change a specific document’s EOL (end of line) by using “Edit -> EOL conversion -> UNIX format”. Another feature that I like is the “Search -> Replace…” which allows you to use regular expressions (like Perl).
Statistical software: R of course! It’s best to do a custom installation and choose a directory without spaces in it. That will help later (further below). If you want to be convinced to join the R community read Data Analysits Captivated by R’s power and R You Ready for R? or just take a look at what others are using. Remember that the R project is open source, free and easy to contribute to. If you end up choosing Excel as your statistical software, well, there is no hope for you!!
R code editor: Notepad++ with NppToR. For a long time I used Emacs modified to work with Windows by Vincent Goulet available here. It works great and saves you quite a bit of setup time. XEmacs is another option that a friend of mine used, but it never convinced me. Anyhow, I ended up changing from Emacs to Notepad++ with NppToR because I could:

Force quit R (and not lose code changes) in case I crashed R by doing something stupid like printing something huge or w/e :P
Access help pages in a separate window. I’m sure you can do it too with Emacs, but I was just lazy to configure it.
Shorter shortcuts
Later on I found the NppToR to PuTTy feature which is very useful.
You can create an R syntax dictionary (or something like that) in Notepad++ which will scan all your R packages and add the function names so they are colored when you type them. Also Notepad++ will auto-complete some function names and show you the arguments. Great stuff! (Forgot the name, so… google it :P)

SSH: PuTTY. As said before, works well with Notepad++ and NppToR.
SCP: WinSCP. There are others that work too like Filezilla but well, WinSCP does the job well.
PDF viewer: Adobe Acrobat Professional. I’m using the X version now. I like how I can highlight, underline, cross out, free hand, sticky note, combine files into a single pdf, combine pdfs, and change the highlight colors easily. It also has a change tracker (kind of like Word has). I’ve seen other use PDF Annotator which is available for free for Hopkins students. Anyhow, I simply love Acrobat for reading papers.
LaTeX: MiKTeX. For writing TeX files I used either Emacs or Notepad++. There is another software which has drop down menus and the like called WinEdt. I got used to typing LaTeX from scratch, well, I have a template.Rnw somewhere. Oh yeah, I always use Sweave when writing TeX files (even if I don’t use R).
R reports: Sweave by Friedrich Leisch, one of the champions of reproducibility! To learn more about Sweave first read this pdf by Nicola Sartori. This is another Sweave demo by Charles J Geyer. Check out this great Windows Sweave troubleshooting page by John D Cook.
Building R packages from source. You will definitely need Rtools installed. I would also install QPDF which can be used by R to compress your pdf files, which is a good thing if you want to have a small-sized tarball. Last but not least, check out Building R packages for Windows by Rob J Hyndman.
Learn to modify your PATH! Check 1.3 from the previous link by Rob J Hyndman. If you are going to use Sweave, it’s best to add to your PATH the path for the directory containing your Sweave.sty file so that you won’t need to copy it to every single directory. This is why it pays off to do an R custom installation and put it in C:/R/R-current-version or something like that instead of C:/Program Files/ bla bla with spaces. It used to be more important a few years ago. Also, I created a sw.bat file and put it somewhere where my PATH would find it. That sw.bat file ran Sweave, pdflatex twice, then bibtex and finally opened the pdf file.
PDF viewer for LaTeX files. I only learnt about pdf sync and the like a year ago. You should google how to set this up with SumatraPDF (Adobe Acrobat doesn’t work!).
Version control: Mercurial. It’s very easy to use and you can get an account at Bitbucket.org with unlimited number of private repositories if you have an academic email. Even if you are not doing a collaborative project, you will love using a version control system! It will clean up your directories very nicely and will help you become more organized. Learning a few commands is nothing compared to having lots of files with _v1 v_2, etc at the end. Check out the Mercurial guide to get started. Note that for windows instead of customizing your .hgrc file you will customize a mercurial.ini file.
Presentations: both PowerPoint and Beamer (normally with Sweave too). Rarely I use Google Docs for this.
Office: Either Microsoft Office or OpenOffice (free).
Poster creator: PosterGenius (academic discount price). It was very easy to use and I would surely give the free trial version a go. It adds a watermark, but well, you will appreciate the time you save compared to using PowerPoint. I guess that Adobe Photoshop is another option, but I’ve only used it to edit photos here and there, not to make a whole poster. The most I did was create this.
To de-compress RAR files: WinRAR.
Anti-spyware: Avast (free version). I normally keep it in silent (gaming) mode so it doesn’t show pop ups.
CCCP codec pack which includes the Media Player Classic. Great for watching video files and dumping the crappy Windows Media Player.

Linux (Ubuntu)

Ubuntu provides Linux distributions that are very user friendly and that look much like Mac OS does now. You’ll find it easy to run multi-core programs which were a pain to do with Windows. Beware that even if you use Ubuntu you will need to learn stuff like how to compile. Also, please check before you install that your computer is supported. For example, some laptops with very new video cards might not work properly. That being said, with Ubuntu you will feel very at ease working in an area like mine (genomics) because a lot of the software runs in Linux (normally in a cluster, but you can test in your lap).

You will want to check and/or keep for reference LINUX Essentials by Thomas Girke (more from him below), Learn Linux in 10 minutes, The Linux tutorial, and Linux vi editor tutorial.

* If you are going to use Linux (Ubuntu) at some point you will want to compile something from source and find out that you are missing a dependency. That’s when I google, then use : 1. apt-cache search something 2. sudo apt-get install something

Note that you will frequently need the yyyy-devel version which includes c headers and stuff that you need to compile.
You will find a lot of things through the package installer (forgot what it’s called). Learn the pseudonym for your Ubuntu version so you select the appropriate version of the software in case that you are downloading it from another place.
SSH/SCP: terminal commands :) I just wanted to mention that rsync is a nice command for synching folders (recursively too) between your computer and say the cluster.
Version control: Mercurial again. The configuration file is .hgrc not mercurial.ini
Text editor: Nedit or Emacs. Vi when doing in-terminal modifications.
You can get R through aptitude or if you want the very latest (or a devel version) you’ll have to compile it. The first time you will have to install plenty of dependencies, but it’s good practice.
Office: go with OpenOffice.
LaTeX: install the texlive distribution. I normally get everything so that later when I’m trying to use a TeX package I won’t have to go install it (which is what MiKTeX does for you in Windows).
Video player: VLC.

Mac

Terminal: iTerm2. Mac comes with a native terminal, but iTerm2 has other nice functions like tabs and more options to customize it. Check this for free color palettes. I like the Homebrew one from here.
LaTeX and R editor: Aquamacs is a version of Emacs that works great. However, as I discussed in the Windows section I’m moving away from Emacs. Well, to be honest, I don’t want to put the time to learn how to customize Emacs properly and do amazing stuff with it like Kasper does. Recently (since June) I’ve been using TextMate. It has this thing called “Bundles” which provides different hotkeys depending on the file you are editing. Meaning that for Rnw files you can Sweave them directly there and for R files you can either send the code to R or to the terminal (much like Notepad++). The one thing is that it is not free BUT there is a 2.0 alpha release available on github that you can compile. This lengthly discussion can be worth reading if you want to know more about the 2.0 version and the future of TextMate. Someone said there that Sublime might be replacing TextMate but I haven’t looked for any R integration in it. Anyhow, I liked how TextMate included an auto-spell checker that recognizes Sweave/LaTeX code from the box :)
Package installer: MacPorts. It’s kind of similar to aptitude from Linux but it’s Mac only. Note that you will definitely need to get XCode.
PDF viewer: also Adobe Acrobat Pro for the reasons mentioned previously.
PDF viewer for LaTeX: TeXShop which I think comes with the MacTeX distribution. It has the forward sync that Alyssa mentions in her blog.
Text editor: TextWrangler. Has several of the functions I talked about in the Notepad++ section like search and replace with regular expressions. It definitely outperforms the native text editor.
SCP: Cyberduck. I haven’t tried others, but it works and I’m happy with it. I also use the terminal to push/retrieve files like I would do in Linux. Same for ssh and Mercurial.
Version control: Mercurial. Note that you might have to add a site key (like bitbucket’s) to your .hgrc file so it doesn’t complain when pushing files.
Productivity: My Little Pomodoro available from the Mac app store. I love it for following the Pomodoro technique (you can use any other timer that you like) which Hilary introduced me to. It works like a charm when you are under stress and need to be productive. After all, I’m prone to escape the stress and distract myself, so this helps me keep my distractions limited. I’ve also found that when I’m stuck in a problem and I take the 5 min break thinking about something else, well, the machine keeps working and when I come back from the break I have a new idea to try out.
Video player: VLC (includes codecs).

Other stuff

Browser: I used to loveMozilla Firefox (it has a nice sync functionality) but I’ve moved to Google Chrome. It’s kind of a shame that Google started to compete with Mozilla, but oh well bye bye 2007. I use Chrome because it works a tad bit better with other Google tools, but that’s it. It also syncs your bookmarks. Both work great and Opera is still my favorite backup browser. I guess anything but Internet Explorer and Safari.
Learning R. I would definitely check Thomas Girke Programming in R page and Frank McCown’s Producing Simple Graphs with R tutorial. I have my own share of R related slides here.
Learning Bioconductor. Thomas Girke again wrote a great resource for learning how to use Bioconductor for analyzing high-throughput sequencing data files. Bioconductor hosts packages for other technologies/problems, so I would also look at it’s own help pages like the Workflows section. I also like Peter’s R programming pages, specially the heatmap section. I have my own share of Bioconductor related slides here.
Learning LaTeX. I learnt the hard way I guess… I learnt by comparing Sweave files and their output and seeing what changed if I modified the code. Nowadays, I would very highly recommend that you first check the How to Use LaTeX short series of exercises/files by Andrew Matchett. The Not so short introduction to LaTeX is a great resource. For very specific symbols, check the Comprehensive LaTeX symbol list.
Using a SGE (Sun Grid Engine) cluster. For some basic commands look here. For the Hopkins cluster, definitely read this. Finally, for running array jobs check this.
LaTeX and math. You should definitely read the wiki books page for this topic. I kept going back to it over my first year at Hopkins when I really needed to learn all this. The theorems page is nice, but not a must. Same for Common TeX/LaTeX errors.
Figures in LaTeX. Here is a basic overview but the wiki books page for the topic is a must check.
Accents in LaTeX. Check this blog post by “Bugs and Solutions”.
Blogging R code: Pretty-R. I haven’t really used it but it surely looks pretty!!
Cloud storage: Dropbox works great and tons of iPad apps have an option to backup to it which works great with my note-taking apps. Google Drive and others are also around.
Paper (biobliography) organizer: Zotero is amazing! I simply love it :) I pull the bibliography from pubmed or the magazine page itself and to avoid any hassle, I have a “papers” folder in my Dropbox where I only organize them by last name. Then if I want to find something, I go to Zotero and look use it’s great search function. I rarely use it to annotate webpages and I hear that it can now upload files to the cloud. Anyhow, I first used it in my Windows/Ubuntu dual setup. You can use it as a Zotero Firefox plugin or as Zotero stand-alone with Zotero Connector (Google Chrome for example). Finally, Zotero can export your bibliography into a BibTex file :)
Blog: Tumblr. Some like WordPress better, but I like how Tumblr is not only a blogging platform but also a social media tool. I’ve written several posts before on how to customize your blog and other blog related tools. But a must in my point of view is to get your RSS feed “burnt” with feedburner. It has lots of interesting tools and is much better than a plain XML RSS feed.
Notes: Notability iPad app. Great stuff and doesn’t blow up on you (aka, doesn’t lose your notes) like NotesPlus did to me. Anyhow, note-taking in my iPad with auto-cloud backups greatly changed my classroom experience. THere are other apps like this one out there.
I use Voice Recorder HD iPad app for recording lectures. It’s useful when you miss something the professor went over quickly and you are trying to understand it later on.
Email: Gmail with keyboard shortcuts enabled. I also use “Canned Responses” from the Google Labs to specify my signature. I have an academic one, one for Mexico, etc.
Send emails later: Boomerang for Gmail.
Calendar: Google Calendar.
Task manager: Google Tasks from within Google Calendar (not from Gmail, which is doable too) with GoTasks in my iPhone.
RSS reader: Google Reader. Works great. By the way, Orbvious interest (below) also works with Google Reader and has a customizable hotkey for it.
Mark pages to read later: Orbvious Interest for Google Chrome. Great stuff! It syncs between computers and you can use Pocket in your iPad to view the links.
Maps: Google Maps.
Video conversation: Skype, Google Hangouts. If I’m going to help someone remotely, then I use TeamViewer which is free for non-profit purposes. With it you can move their mouse, which makes things much easier for support issues!
Photos: Picasa. I pay the 5 bucks a year for 20 GB on Picasa Web Albums so all my photos are on the cloud.
Dictionary: die.net
Network visualizer/analyzer: Cytoscape.
Venn diagrams with more than 2 sets.
Setting up your website. I pretty much followed Alyssa’s instructions and got my CSS template for my academic page from FCT. Then I used simple html to modify it.

I pretty much dumped a ton of my bookmarks in this huuuuge post! Well, I hope that it will be useful to someone. At least now I’m happy to have contributed to Hilary’s computing-resources-post drive.

The new visualization package for genome data in Bioconductor: ggbio

Tue, 06 Dec 2011 00:00:00 +0000

It’s been a while since I’ve been waiting for the release of a visualization package in Bioconductor. Back in 2008 I was really impressed by the power of GenomeGraphs and I have used it in multiple occasions. Yet from both the Bioconductor Developer Meeting in Heidelberg 2010 and BioC2011 I’ve been waiting for the release of the visualization tools developed by Michael Lawrence and Tengfei Yin at Genentech.

So, after a long hiatus where I didn’t browse the biocviews in Bioconductor, I found out that Lawrence and Yin released ggbio and biovizBase (it’s more of an infrastructure package for ggbio) . I haven’t really had the time to play around with them, but it’s definitely worth exploring both of their vignette files: ggbio, biovizBase. I also think that they’ll fit very well in Bioconductor because quite a few of their examples involved the gamma of objects the BioC team has released for high-throughput sequencing (HTS) data. Meaning that they work well with objects from IRanges and GenomicRanges. Also, some of the examples use BAM files which are common nowadays in any HTS analysis pipeline. As a plus, ggbio uses ggplot2, which definitely makes clear nice plots.

I expect ggbio to replace GenomeGraphs soon (although I love using it), but I’m also kind of disappointed that I didn’t see any of the cool examples from BioC2011 in ggbio’s vignette file. After all, visnab looked pretty impressive as you can see in this presentation. I don’t know if they decided to rename visnab into ggbio, or maybe they haven’t released visnab yet. Anyhow, give ggbio and biovizBase vignette files a look :)

Introducing Biostatistics to first year LCG students

Sun, 04 Dec 2011 00:00:00 +0000

Around two weeks ago I gave a talk via skype to the first year students from the Undergraduate Program on Genomic Sciences (LCG in Spanish) from the National Autonomous University of Mexico (UNAM in Spanish). The talk was under the context of the Introduction to Bioinformatics Seminar Series whose goal is to familiarize the new students with the bioinformatics world. It used to be a course heavy on exploring database websites, some basic theory, and lots of new concepts and algorithm names. Like, what is BLAST? This year, the course involved several talks from former students (like myself) on their experience and current job (most of us are in graduate school).

In my case, I was invited to talk about Biostatistics and R as I was one of the first LCG students to learn and teach R to other students (including PhD students ^^): 12 hour intro to R and Bioconductor, R in an intro to statistics course, a full course on Bioconductor.

I had a lot of fun preparing my talk as I tried to portray the three main currents in Biostatistics in a way that would be understandable, basic concepts such as a P-value, some basic R code (the students knew the super basics only), and doing so in a way that I would also pass how I see things. A key part for any bioinformatician, as I see it, is to have a good basic toolset. That’s why I tried to pass on many tips to these young students. Also, I do like to go back to the philosophy of science and whether we need hypothesis nowadays or just models based on the data. I definitely had to cover the topic of communication as I strongly believe that any researcher has to be able to communicate with a biostastician if they want their help in analyzing their data. Also, a biostatistician is not a stastistician, meaning that they have to understand the underlying biological question. I tried emphasizing this point with the students and I attempted to motivate them to take a basic stastistics and probability course (2 preferably). After all, biostatistics is key nowadays in science since biology has gone high-throughput.

Giving the talk through skype was definitely a new experience for me. The best is to be in the classroom as you miss a lot of the interaction with the students. For instance, I didn’t always know who as asking the question and frequently I had to ask them to repeat it louder. Plus, I chose to share my desktop with them instead of having them watch at my face all the time. I guess it wasn’t easy for them to just watch a screen with a background voice for 2 hours :P But well, I hoped they liked it as much as I liked preparing it.

If you are interested in the talk, I uploaded the PDF file, the associated R file and the master Rnw file to Google Docs. The Rnw file can be useful if you are learning how to write Beamer presentations using Sweave and LaTeX.

Finally, I was asked to prepare two short questions so that the teachers can evaluate the students. As a bonus for any of them reading my blog ^^ (I did portray blogs as an excellent way to update yourself) here they are:

Argue why you need to learn (at least basic) Biostatistics and why it’s useful nowadays to analyze biological data.
Based on the data from your recent experiment, you construct a 95% confidence interval (frequentist approach) for the mean of your variable of interest. What is the probability that the true mean of the distribution is contained in the confidence interval.
Bonus: list 5 basic R plotting functions

Answers:

You need to learn the basics (at least) of Biostatistics in order to be able to communicate with your biostatistician collaborator, and to analyze your own data specially by performing exploratory data analyses (EDA). Also, biological experiments nowadays generate a lot of data as biology has gone high-throughput. Biostatistics is specially useful in order to analyze all this data.
0 or 1 and you cannot know which is the case for your specific dataset.
I mentioned: plot, hist, boxplot, qqplot, qqnorm, lines, points, abline, legend

Fellgernon Bit - rstats

Cleaning up my R packages and config files

Config files

Wrapping up

Acknowledgments

References

Reproducibility

The start of a new phase for me

Acknowledgments

References

Reproducibility

Research Scientist: an academic career launch pad

What will I do as a Research Scientist at LIBD?

Details

Final decision

Acknowledgments

References

Reproducibility

Diving together into the unknown world of spatial transcriptomics

Spatial transcriptomics

Our project’s history

spatialLIBD

Study design

Shiny website mirrors

R/Bioconductor package

Installation instructions

Access the data

Processed data

Citation

Closing remarks

References

Reproducibility

Learning from our search history

Origin of the idea

Motivation

The theory

In practice

We are diverse

Learn from each other exercise

Some rules

Main steps

Goals

Test session

My search example

Conclusions

Acknowledgments

References

Reproducibility

Conference feelings: from newbie to sponsor

Why I like going to conferences

As a newbie

As a… sponsor?

Feeling empowered

Acknowledgments

References

Reproducibility

Windows user space issues with installing R packages

Solutions

Backstory

Uncompressing

Installation

clusterProfiler installation

Closing

Acknowledgments

References

Reproducibility

The evolution of my academic career as seen through posters and talks thanks to hugo academic 4.1

The end result: alumni

My academic career

Some code for udpating to hugo academic 4.1.0

Acknowledgments

References

Reproducibility

CDSBMexico: remember to apply for BioC2019 travel scholarships

BioC2019

Travel scholarships

Get started on your scholarship application!

Website

Bioconductor project participation

Benefitting from BioC2019

`spatialLIBD`

`clusterProfiler` installation

New to `blogdown`?