Posts | L. Collado-Torres

Initial impressions from testing Posit Connect

Fri, 14 Mar 2025 00:00:00 +0000

Two weeks ago I finished testing Posit Connect as a replacement to shinyapps.io. At the end of the trial period, I presented my conclusions during a LIBD rstats club session. Here you can read the detailed notes and watch the resulting video:

The video and notes dive more into the technical details and I also performed some live tests. Below I summarize the main points from our use case and solutions we have experience with.

Our use case

Overall, my team and colleagues at LIBD are in a position where we have:

Over 75 iSEE and spatialLIBD web apps powered by shiny and hosted on shinyapps.io.
- These apps have been deployed by 14 different people so far. That is, we have been able to teach new members or students how to deploy apps: anyone can do so with a bit of training.
Our apps are typically shared publicly, so we don’t really need password-controlled access.
- We don’t share URLs for studies under development assuming that no one will correctly guess the URLs.
Our data is stored in SummarizedExperiment, SingleCellExperiment, or SpatialExperiment objects. The single cell and spatially-resolved transcriptomics data typically involves over 50,000 cells or Visium spots across 20,000 or so genes. Some larger studies are over 100,000 or 200,000 cells/spots. The cells/spots are the columns and the genes are the rows in these large matrices of expression data. Many cells/spots don’t have expression data, so the matrices are sparse. For more on these types of objects check the video below.

For spatially-resolved transcriptomics data, SpatialExperiment objects also contain images per sample (defined as a Visium capture area). These are typically lowres images that are maximum 600 x 600 pixels. One of our largest studies has 120 samples.
- There are other images that we typically remove from the objects, such as this case which saved 3.29 GB of memory by dropping non-essential images in a study with 36 samples. The initial SpatialExperiment object used 8.81 GB in memory for this study.
The expression matrices are typically saved as dgCMatrix objects from the Matrix package. These are great for storing sparse data if you want it all in memory. However, for our larger studies, we typically just keep one matrix that has the library-size normalized counts (called logcounts) and drop the un-normalized counts.
- In the same dataset with 36 samples, this meant another 2.6 GB saved in memory as shown here. This resulted in a reduced SpatialExperiment object of 2.92 GB in memory for this study containing only the logcounts normalized gene expression data and the lowres images.
For larger studies, we can use HD5Array objects to store the expression data on disk and only load the data in memory when needed.
- For a dataset with 120 samples, this results in a 505.20 Mb SpatialExperiment object in memory that has the counts, lowres images, 36,601 genes, and 599,034 spots as shown here. The files on disk use about 1.5 GB of space: 1.4 GB for the .h5 file as noted here.
- Actually, for this study, we have another SpatialExperiment object with logcounts and counts, no images, 28,965 genes, and 535,248 spots that uses 154.04 Mb in memory as noted here. However, the files on disk grow to use 13 GB of disk space for the .h5 file as shown here. I believe that’s because the logcounts which have decimal numbers take much more disk space to store them. This same object as a sparse memory object and with the lowres images added used 10.69 GB of memory as noted here.
- HDF5Array is awesome for saving memory, but it can take up a lot of disk space. It’s a great resource developed by Hervé Pagès from Bioconductor’s core team.
- For more on HDF5Array and compressed data, check this video:

We have observed that iSEE and spatialLIBD need more memory than just the one needed for loading the object as reported on the memory usage graphs provided by shinyapps.io. That’s because of some internal operations, and while both packages try to minimize memory duplication, it’s best to have more memory than twice the object size to prevent the apps from crashing: if they are unreliable, people won’t use them.
- In practice, we try to deploy objects that use up to 3 GB of memory when using the 8 GB memory instances on shinyapps.io.

Use case summary

Overall we have been able to use shinyapps.io for studies with R objects that use up to 3 GB of memory with sparse memory dgCMatrix objects for storing the logcounts normalized expression data. In the case of spatially-resolved transcriptomics projects, we only include the lowres images. These objects can also be uploaded within the 3 to 5 GB file size limit as noted in the official shinyapps documentation. However, we have datasets where this is no longer possible. You either need more memory or the ability to save data in files larger than 5 GB.

Deployment options

We have considered the following options for deploying our apps:

AWS-hosted shiny-server
- You choose how much disk space and memory you need on a given AWS instance
- You have to install R yourself and the open source shiny-server
- You can only have 1 app deployed by AWS instance
- You upload the data and R code for the app yourself
- For us, a LIBD system admin has dealt with the AWS and shiny-server setup, plus the R updates. Then a second person (myself so far) has had to upload the data and keep the R code up to date. In practice, we rarely update this app.
- Example: http://spatial.libd.org/spatialLIBD/ which at some point was deployed on an AWS instance with 128 GB of RAM. We wanted to make sure that this app could handle several dozen users.
shinyapps.io
- If you have the best account (Professional; check their pricing information) you can have apps deployed with up to 8 GB of memory and max 3-5 GB disk space.
- For our apps, we typically configure them to have 1 R process per instance, with 2 max connections per instance. Aka, up to 2 users per instance.
- Each app can run in up to 10 instances, so you can have up to 20 users per app.
  - You can deploy the same R code and data under two apps (aka two URLs) or more if you want to have even more capacity.
- Deployment is done via rsconnect::deployApp() which has to upload your data every time and install packages too. While compiled R packages have increased the speed of this process, it can easily take 15-30 minutes per app.
- Example: libd.shinyapps.io/spatialDLPFC_Visium_Sp09 from the spatialDLPFC project.
Posit Connect
- You can upload data to the server separate from the R code for the apps.
  - This requires ssh access, but that’s fairly easy to implement.
- It can auto-sync the R code from GitHub repositories every 15 minutes.
- It can auto-sync changes to the list of R packages also through GitHub repositories. This involves manifest.json files made with rsconnect::writeManifest().
  - It makes it incredibly easy to update many applications at once by just updating the manifest.json files.
  - Enables a much faster update process than shinyapps.io as we make updates to spatialLIBD to add new features or fix bugs.
- Your server can have 200 GB of disk space and 31 GB of RAM (as the one I tested) or more resources.
  - This unlocks the ability to upload HDF5Array-compressed data that has less memory requirements but larger disk space requirements.
  - Note that the server resources are shared among all apps on the server!
    - My understanding is that Posit Connect has ways to deal with resources running out on a server, but I haven’t tested this feature.

Conclusions

Overall, I’m hopeful that we’ll get long term access to a Posit Connect server. It will allow us to deploy our apps with larger datasets and more memory. It will also make it easier to update our apps and keep them in sync with our GitHub repositories. This will be especially useful as we continue to develop new features and fix bugs in our apps. For example, I typically test new spatialLIBD features on one dataset. But we now have many datasets each with different properties, and it’s hard to test them all. Instead, with Posit Connect, I could modify spatialLIBD, then easily update several apps on Posit Connect, and check them to see if the new features work as expected. This will make it much easier to maintain our apps and keep them up to date.

If I were to start today, I would use the organization that I tested at github.com/LieberInstitute/Posit_Connect_shiny_apps using only HDF5Array objects to take advantage of the memory savings and faster app load times you get with those objects compared to dgCMatrix objects (it takes several dozen seconds more to load a ~10 GB object compared to a 500 MB one). With just these changes, I think that our users will be happier with the apps. Simply being able to visualize our largest datasets would be a great win for us.

Of course, I don’t have pricing information for Posit Connect yet. But we have reached the limits of what the best shinyapps.io accounts offer. I believe that other options exist, but I haven’t tested them yet. Though from what I’ve seen, Posit Connect works and I’m very confident that we could teach our team members and collaborators on how to use it effectively.

Regardless of where our apps are deployed, we also need to work on a way to document them all and make it easier for others to find them. I’m aware of iSEEhub’s existence which makes an app that launches iSEE apps from data hosted by ExperimentHub. We might need to make something like an spatialLIBDhub for our datasets.

PS. The two members from Posit with whom I interacted with for this test were very friendly and helpful. Thanks again!

Teaching omics methods to LCG-UNAM students

Fri, 07 Mar 2025 00:00:00 +0000

Notes from a second semester introductory class to the world of omics research techniques at LCG-UNAM. That is, my undergrad alma mater. Thank you for the invitation!

Valentina Arias Ojeda is as teaching assistant for the class “Omics Research Techniques”, taught by Dr. Constance Auvynet at LCG-UNAM. They invited me to teach a class on single cell RNA-seq and spatially-resolved transcriptomics. Valentina introduced me and then I mentioned a bit of my teaching history, starting from my first teaching experience where I made a website available in both English and Spanish.

I decided to teach today’s class as a collective “Data Science guidance session”. However, unlike a regular DSgs, we first needed to come up with a list of questions or concepts that they wanted to learn more about. Here I used a technique I learned as a teaching assistant for the 140.620 series on Statistical Methods in Public Health at JHU. That is, write on the board the questions, then try to group them up into different categories. Our full list of questions was:

RNA-seq
- What is the difference between microarrays and RNA-seq?
- What is RNA-seq?
- How do we deal with ribosomal RNAs in RNA-seq?
- How do we deal with noise in RNA-seq?
  - How do we define noise?
- How does RNA-seq deal with RNA secondary structures?
Single cell RNA-seq
- What is scRNA-seq?
- What is scRNA-seq useful for?
- Which are the main sequencing technologies?
- What is ATAC-seq?
- What is insitu seq?
Spatially resolved transcriptomics
- What is spatially resolved transcriptomics?
- What is the difference between spatially resolved transcriptomics and scRNA-seq?
- How can we integrate RNA-seq and spatially resolved transcriptomics?
- What are the physics behind spatially resolved transcriptomics?
Data analysis
- Why do we use R for omics data analysis?
  - How do we analyze RNA-seq data?
  - How do we analyze scRNA-seq data?
  - How do we analyze spatially resolved transcriptomics data?
- How much computational power do we need for spatial omics data analysis?
- How do we integrate RNA-seq and spatially resolved transcriptomics data?

RNA-seq

I used this slide with an image from Gandal et al DOI 10.1016/j.biopsych.2020.06.005 that shows many types of RNA molecules. Many are mostly interested in mRNA: messenger RNA. Though other RNA molecules can be of interest, such as microRNAs.

For microarrays, I explained how they are probe-based assays where a company designs the probes, prints them on a glass screen, and how the data is generated through high resolution images (photos) of the glass screen. To do this, you have to know the gene annotation (where genes are located in the genome) in order to design the probes. Microarrays are cheaper than RNA-seq and have been used a lot for DNA genotyping, where we are interested in a set of 2.5 million or so locations in the genome that commonly vary between individuals.

RNA-seq really came to be popular or well, possible, with the advent of next generation sequencing. The idea is that you can sequence the RNA molecules in a sample and then align the sequences to the genome to know where they came from. This is more expensive than microarrays but it has the advantage that you don’t need to know the gene annotation beforehand. This is because you can align the sequences to the genome and then count how many sequences align to each gene. You can also study isoforms (transcripts), instead of genes. A quick Google search led me here as I wanted to show them an illustration of a gene with multiple transcripts, some of them with very small differences between them. We also looked at Gencode’s release history to highlight that the gene annotation is not static and frequently updated.

For a quick intro to next generation sequencing or high-throughput sequencing (these were synonyms in 2006 to 2011 or so), I went back to a course I taught for UNAM PhD students shortly after finishing my undergrad. From https://lcolladotor.github.io/courses/PDCB-HTS.html, we looked at the slide deck for “2. Understanding the technology”. Particularly slides 39 to 42 where I briefly explained Illumina short read sequencing.

In relation to ribosomal RNAs, I explained the two most common RNA-seq protocols: polyA+ and ribo-depletion (RiboZeroGold). I used a preprint we wrote in 2024 where Figure 1B shows the two RNA-seq library types but also the different RNA extraction protocols (often something people ignore).

Figure 1D highlights how different the data using principal components (PC1 separates library type, PC2 separates RNA extractions), and Figure 1E shows within a given RNA extraction, how genes are differentially quantified by the two library types.

For noise in RNA-seq, we looked at my derfinder PhD paper. Figure 1 actually is a reminder of how transcripts can have very small differences between them. Figure 2 has coverage curves for RNA-seq data. We used derfinder when we studied human brain expression changes across age groups including prenatal samples DOI 10.1038/nn.3898. Through that work we concluded that the human brain transcriptome was incomplete. I mentioned also how polyA and RiboZeroGold data will have different coverage profiles (RiboZeroGold will show coverage for introns as some pre-mRNAs are going to be sequenced). I also described the totalAssignedGene metric that SPEAQeasy generates: the fraction of reads assigned to genes from the total number of reads that were aligned to the genome. We’ve used totalAssignedGene to identify poor quality samples in some studies.

I also briefly mentioned that an advantage of RNA-seq is the possibility of assembling transcripts using tools like StringTie developed by some of my collaborators. They ran StringTie on the ~10,000 GTEx RNA-seq samples to build CHESS. Eventually from ~26 million transcripts they filtered the list and found ~160,000 transcripts DOI 10.1186/s13059-023-03088-4. There are many challenges on how you decide what is a real transcript versus noise. Do you trust more a transcript with 100 reads but only in one sample or a transcript with 5 reads per sample that is present in 10% of your samples?

As for secondary structures in RNA, I explained how RNA-seq protocols involve a step where RNA is cut into fragments of about 300 to 500 base pairs. That’s one way to deal with RNA secondary structures. Though I don’t remember if there are other steps that help avoid issues with RNA tangling. And it might be something that we forget about: some RNAs that are more likely to fold might not be as well represented in RNA-seq data.

scRNA-seq

For scRNA-seq, I used parts of this explanatory video from 10x Genomics. I paused it a few times to highlight the GEMs, the fluidics for getting cells into droplets, and how each droplet has a cell barcode (a unique DNA sequence), and there’s about 750,000 of them on a given experiment.

In the end you generate DNA that you can sequence with Illumina sequencers. Actually, many other protocols are all about generating DNA for the molecules of interest, then sequencing those DNA molecules with Illumina. For example, ChIP-seq, ATAC-seq, etc.

Spatially resolved transcriptomics

To explain spatially-resolved transcriptomics, I showed the students my blog post (with English and Spanish videos) explaining the first 10x Genomics Visium paper which we wrote DOI 10.1038/s41593-020-00787-0. I highlighted how Visium uses “spatial barcodes”, akin to the scRNA-seq barcodes we saw previously. These DNA sequences label the nearly 5,000 spots present on a Visium slide. Here’s a short 10x Genomics intro to Visium video.

For the tissue we were studying (human brain DLPFC), we found that each Visium spot contained on average 3 cells. With spatial omics you find how cells are grouped spatially, which can be informative unlike what we obtain with scRNA-seq (individual cell expression profiles with no spatial information) or bulk RNA-seq (like blending together a bunch of cells and getting the overall mean expression profile). Spot deconvolution, which we benchmarked a bit at DOI 10.1126/science.adh1938, and bulk RNA-seq deconvolution which we’ve also benchmarked at DOI 10.1101/2024.02.09.579665, are some computational methods that allow integrating multiple flavors of RNA-seq together. Dai et al also wrote a really nice benchmark paper for second generation bulk RNA-seq deconvolution methods using scRNA-seq data as the reference: DOI 10.1126/sciadv.adh2588.

I also highlighted how with bulk RNA-seq some LIBD datasets have 80 million paired-end reads of 100 base-pairs each (as about $1,000 USD per sample). Ultimately, the read length (in base pairs) multiplied by the number of reads determines how much money you are doing to spend sequencing. 10x Genomics recommends 50,000 reads per cell for scRNA-seq or per spot for Visium. A Visium capture area has 5,000 spots, aka, that’s 250 million reads per Visium capture area. You do spend about 2,000 to 2,500 USD per Visium capture area just for Illumina sequencing without including other 10x Genomics reagents. Also, the options you have per cell or per spot are limited as you went from 80 million reads to 50 thousand (a 1,600x reduction!). So scRNA-seq and Visium are much more sparse than bulk RNA-seq. Some scRNA-seq protocols are also 3’ biased, meaning that you only sequence the 3’ end of the RNA molecules. Some have developed ways of quantifying transcripts despited these biases, but it’s still an active area of research. Overall though, the data generation costs for each assay sometimes determines what assays one can use. However, if others have shared relevant data, then you can re-use that data and don’t have to generate all yourself.

In terms of the best methods for RNA-seq sequencing and associated assays, I mentioned that one way to check this is by looking at the valuation of these companies and their stock prices. These companies frequently acquire smaller companies, sue each other, etc. Some of them have been impacted by recent trade wars, for example China bans imports of Illumina’s gene sequencers right after Trump tariff action. This was partially done because two genomics Chinese companies were accused by the US on grounds of national security. Overall, Illumina and 10x Genomics are among the biggest players in the field. There are other large companies, and many small ones (20 or so employees), some of which aim to be acquired by the big ones.

Data Analysis

I’ll see these same second semester students a year from now when they take the fourth semester class on introduction to RNA-seq data analysis with R and Bioconductor that I’ve taught the last five years. As a quick teaser, I mentioned that Bioconductor is a great repository of open source software for the analysis of high throughput biological data: microrrays 23 or so years ago and nowdays scRNA-seq and spatial omics. It’s also mandatory to write user guides (vignette documents) on all Bioconductor software. This makes it much easier to learn how to use the software and it’s a distinctive factor that separates Bioconductor from other software repositories.

From my team’s recent TL DR (too long, did not read) presentation I showed a few slides.

Check our recent video "[2025-01-29] R/Bioconductor-powered Team Data Science: TLDR version 2025" youtu.be/alL-g2roXxM
We go over what analyses our @lieberinstitute.bsky.social has expertise on 💻

Slides orchestrated by @lahuuki.bsky.social at speakerdeck.com/lahuuki/2025…

👥 go.bsky.app/53gy4UP

[image or embed]
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor.bsky.social) February 12, 2025 at 3:35 PM

First I showed how Visium HD is a new technlogy from 10x Genomics that is probe based. So think about it as a modern microarray that is spatially-resolved. It’s also powered enough to generate sub-cellular resolution data: starting at 2 micron squares.

Analyzing that data can be computationally demanding in terms of requiring GPUs and/or high-memory compute nodes.

Ultimately, there can be a need to uniformly process publicly available RNA-seq data to democratize access to this data such that researchers without access to high performant compute environments are also able to analyze these valuable datasets. I worked on this through the recount2 and recount3 DOI 10.1186/s13059-021-02533-6 projects. I’ll teach them more about this next year.

Miscellaneous

I ended with highlighting the resources from the Community of Bioinformatics Software Developers (CDSB in Spanish) and the LIBD RStats Club. There students can find videos from full week-long workshops such as the 2021 scRNA-seq analysis course and many other videos from LIBD RStats Club schedule.

I also highlighted how one has to spend work hours trying to learn new methods instead of relying on doing so on your free time. Finally, I highlighted how I like to learn in public. How sharing what you know can be helpful not just to others, but yourself.

It’s always great to be back at LCG-UNAM, my undergrad alma mater. I also enjoy interacting with current students. Here I’m with Valentina and Bernardo, both of whom are working as teaching assistants and finishing their third year. Next up is their final year which is a research year. I’m looking forward to learning about that they end up choosing to work on and to witness their careers flourish.

Here are all the links that I showed in class today.

Genomics scientists from and in Mexico community building

Mon, 11 Nov 2024 00:00:00 +0000

I recently gave a remote seminar presentation for the Applications of Genomics 2024-2025 course organized by Esperanza Martínez Romero and Alejandra Zayas Del Moral. The idea was to talk about my career and showcase some recent research work from my team.

A week ago I had the privilege to present to #LCG-UNAM students as part of the Aplicaciones de la Genómica course by Esperanza Martínez and @alezayas.bsky.social
www.youtube.com/watch?v=a_Ip… (in Spanish 🇲🇽) 🛝 speakerdeck.com/lcolladotor/…

#DeconvoBuddies @bioconductor.bsky.social #RStats

[image or embed]
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor.bsky.social) November 7, 2024 at 11:07 AM

But I also wanted to take the opportunity to talk about the importance of community building and relying on the support from your communities as you advance in your career. I’m from the third class of LCG-UNAM undergraduate program on Genomic Sciences from the National Autonomous University of Mexico (UNAM) and graduated in 2009 (see my timeline blog post for more details). That means that we didn’t have as many people to look up to as you do now. Here’s a short extract of how I relayed some of this to the students at the end of my seminar.

Note that if you don’t speak Spanish, YouTube can auto-translate the captions to your language of choice ^^.

LCG(EJ)-UNAM network

Let’s say that you are a current student and want to tap into the LCG(EJ)-UNAM network. Well, the LCG-UNAM and LCG-EJ-UNAM network is actually not that easy to find 😔.

Note that the two sibling undergrad programs are actually run by different people, both from UNAM. I personally consider them to be part of the same group (particularly from an international lens) as I think that diving them is not helpful. Though I understand that each program has specific areas they want to emphasize, different funding issues, and different challenges they face.

Emails

Ok, so maybe one way to get in touch with others is via email. For LCG-UNAM there’s a mailing list for everyone, but that only includes @lcg.unam.mx emails which many alumni don’t check/use anymore. For LCG-EJ-UNAM I’ve been told that there’s no mailing list.

For each class in LCG-UNAM there’s also another mailing list, but well, they have the same issue of only including @lcg.unam.mx emails.

During the lead to LCG20 celebrations (20 year anniversary of LCG-UNAM), a Google Sheet was spread around where people wrote down their name, class, and email. So I’ve also used that email list (many of which are @gmail.com accounts) too. There are 130 emails listed here (some with typos!).

Facebook

LCG-UNAM alumni have set up a private Facebook group. It currently has 355 members. I see that some members are LCG-EJ-UNAM alumni, so I believe that both programs are welcome there. This is probably the one resource most people use when trying to ask for help or share opportunities. However, many don’t use Facebook anymore, so I wonder how many of the members are actually active. It does have the benefit of being a private group, unlike other social media platforms. So I think that this Facebook group will continue to be one of the main ways to get in touch with LCG(EJ)-UNAM alumni.

Slack

As you might know, I’m a big fan of Slack for communicating for work. Yet, as far as I know, there’s no LCG(EJ)-UNAM Slack workspace. However, as co-founder of the Community of Bioinformatics Software Developers, I’ve been encouraging people to join us at the CDSB Slack workspace.

One way I do so is by asking students from the LCG and LCG-EJ courses I have taught to join the CDSB Slack where I set up a channel for the course and communicate with them through it. Then once the course is over, they are part of the CDSB Slack and can network with others there. This Slack workspace isn’t as active as I would like it to be, but well, it’s a start. It currently has 396 members on the #general channel.

X/Twitter and now Bluesky

During my courses I have also encouraged students to create a Twitter account to be able to network with other scientists at large, get their name out there, and meet people who can help them along the way. Maybe indirectly, just by sharing interesting resources. Maybe directly, by asking each other questions and then possibly meeting up in person later on. Some LCG-UNAM alumni historically were very active on Twitter.

Bluesky has taken off

However, X/Twitter isn’t what it used to be. In recent weeks, the open-source alternative Bluesky has taken off. In 2023 you had to get a code to join, which made it really unfeasible to get students to join or the community at large. But things are different now at the end of 2024 and Bluesky has really taken off lately. One of the reasons is starter packs, which allow anyone to recommend up to 150 accounts that others might want to follow.

One personal reason why I had still being using X/Twitter was because I felt that my community of genomics scientists trained in Mexico, working in Mexico, or helping train Mexicans wasn’t represented in Bluesky. I appreciate that I was included in the “RStats Starter Pack” by Jeremy Allen, which has really helped with my visibility. As I’d like to make the full switch to Bluesky, I’m using my new found visibility to try to encourage others to make the switch as well.

Genomics scientists trained/working in Mexico 🇲🇽

To try to implement my wishes, I created a Bluesky starter pack. The name is limited to 50 characters, but well, I wanted to make it broad in a sense: anyone who has been trained in Mexico, works in Mexico (don’t need to be a Mexican citizen), or has helped train Mexicans within the genomics scientific domain.

This definition definitely includes LCG-UNAM and LCG-EJ-UNAM members, but also CDSB, RMB, and NNB-UNAM workshops alumni. Plus scientists working or involved with CCG-UNAM, IBt-UNAM, LIIGH-UNAM, among other institutions. Yes, UNAM is highly prevalent right now for many reasons, but one of them being that I was trained at LCG-UNAM and thus know more about that network right now.

Getting people to join Bluesky

At first I found close to 30 Bluesky accounts for the starter pack. Most of them I knew of already since we had been sharing Bluesky codes via WhatsApp and other media in 2023 to try to get people to join Bluesky. To get more, I wrote about the starter pack to the @lcg.unam.mx mailing list, the list of 130 LCG alumni emails, as well as posted about it on Facebook, X/Twitter, CDSB Slack, and LinkedIn. And of course, Bluesky itself.

But I know that a mass message like that is only going to get so much attention. So I applied a thing I learned from Alejandro Reyes while working together in the first years of CDSB’s existence. That is, contact people directly.

I contacted 147 X accounts via DM and 9 via public posts from people I follow on X/Twitter. I also emailed all the organizers of RLadiesMX since I think that it would be great for them to also join Bluesky. I also need to contact the communities listed at on my JHU Statistical Computing course website. I also texted some via WhatsApp since they have Bluesky accounts that are inactive, and thus I cannot add them to the starter pack.

I’d say that between a third to a half of them have replied so far. Many ask: What’s Bluesky?. Others had heard about it about but had no invitation code to join in 2023 and didn’t know that you no longer need a code. Others are hesitant to start an account in a new social media platform. But most are eager to join and soon after have created their Bluesky account (some even had one that I missed!).

Currently we have 77 accounts listed on the starter pack out of the max 150. I’ll probably have to make a part 2, or maybe even a part 3 starter pack later down the road. But well, I’m excited to see more of us join Bluesky. After all, I want our voices to be heard and it’s hard to do so if we are not in the conversation to begin with.

Just one weekend of time can make a huge different in community building. But well, once you get the ball moving a bit, it’s up to the community to take it from there. I’m excited to see where this goes and how we can help each other out in the future 🙌🏽 🇲🇽 🧬 ^_^.

HumanPilot: first spatially-resolved transcriptomics study using Visium

Thu, 23 May 2024 00:00:00 +0000

Would you prefer a video walkthrough over reading this blog post? Check out this explainer video 🎥

A few years ago now (2021) we published a study we refer to with a very generic name: HumanPilot (Maynard, Collado-Torres, Weber, Uytingco et al., 2021). In this peer-reviewed study, we were the first ones to use a commercially available technology for generating spatially-resolved transcriptomics data. It’s called Visium and it’s sold by 10x Genomics. Note ⚠️ that members of that company were co-authors of our study. As we work in the Lieber Institute for Brain Development, we tested Visium with postmortem human brain data. I explained the initial pre-print (before peer-review) version of our work in a blog post back from Feb 29, 2020.

🔥off the press! 👀 our @biorxivpreprint on human 🧠brain @LieberInstitute spatial 🌌🔬transcriptomics data 🧬using Visium @10xGenomics🎉#spatialLIBD

🔍https://t.co/RTW0VscUKR
👩🏾‍💻https://t.co/bsg04XKONr
📚https://t.co/FJDOOzrAJ6
📦https://t.co/Au5jwADGhY https://t.co/PiWEDN9q2N pic.twitter.com/aWy0yLlR50
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) February 29, 2020

But enough with the fancy specific words. Here I explain the main points of why this study was so important and why it is still frequently cited several years later.

What is a fruit? Banana, orange?

When we are growing up, we don’t know what is an orange or a banana. What makes them both fruits, and why they are different.

If you look at them, touch them, smell them, you start to notice differences between them. In science, we end up trying to come up with systematic and specific ways of describing things that surround us in nature. In this case, two fruits. When you look at an orange and a banana and try to explain how they are different, you might end up slicing them.

Once you slice them, you are actively zooming in and looking at finer differences. You might end up using a microscope to notice even smaller differences.

Measuring gene activity

Now imagine that you want to measure gene 🧬 activity levels. Some previous technologies could measure gene activity levels by taking a chunk of tissue to have enough material for the technology to work. In the 2010s the technology was developed to measure single cells or nuclei, but you lost the spatial context. Then in 2020 the ability to measure gene activity while retaining the spatial awareness was crowned the new method of the year. The specific term is spatially-resolved transcriptomics.

Visium in particular works by having about 5,000 circles (called “spots”) arranged in a honeycomb pattern 🐝 where we can measure gene activity levels. 10x Genomics has some videos explaining how to it works, but it basically comes down to having an X and Y map for which you can make multiple 3D barplots like this one. Say that this is the one for gene 1 where we have a flat map and the height of the bars is how much that gene 1 was active in each square.

With Visium we have 5,000 spots, or specific (X, Y) pairs and for each of them we can have about 2,000 genes measured. So imagine having 2,000 of these 3D barplots. That’s quite a bit of data and it’s exciting to think what this technology can be used for.

Human brain

Just like we were talking about bananas and oranges being two types of fruits, well, the brain has different regions. Each of them carries out specific functions and can be quite different despite them all clearly remaining a part of the brain. Just like bananas and oranges look very different but still remain fruits.

Now, scientists by training are skeptical. So with a new exciting but expensive technology, the first thing we wanted to do is to test it with a small study. That is, run a pilot study. We work studying gene activity levels from human brain samples, particularly postmortem human brain samples that were donated to our institution for research purposes. One particular region we are interested in is called the DLPFC and its important because differences between individuals that do not have schizophrenia (which we refer to as the neurotypical control group) and those that do schizophrenia have been linked to this brain region, among other reasons.

But before we can distinguish the two groups of donors at the molecular level, with the ultimate goal of helping those affected by disorders such as schizophrenia, we first needed to understand what is typically different neurotypical control (NTC) donors. Just like we can look at different healthy bananas and notice how they are different.

And so we got started with our HumanPilot study. We generated data from 3 NTC donors as you need to look at more than one donor to have an idea of biological variability.

But because Visium was a new technology then, we also generated spatially-adjacent replicates (two pairs per donor). These are technical replicates in the sense that we took contiguous slices of tissue. Think of it as two paired slices of bread 10 microns apart: we generated a pair early in the loaf and one more 300 microns apart in that same loaf. Yes, things do change even when you look at things 10 microns apart, but hopefully they don’t change too much then, but should once you look at them 300 microns apart. That’s what we wanted to see.

The DLPFC has been quite well studied and it was well known that gene SNAP25 is more active in neurons while MOBP is much less active in neurons. When we looked at the activity levels of these genes and could see basically how they complemented each other like a Yin and Yang symbol ☯, we were very excited.

Yin and Yang

We could also see more specific gene patterns, such as PCP4 which marks a specific subsection of DLPFC called Layer 5. And very importantly, this pattern was consistent among the 6 pairs of spatially-adjacent replicates we generated data for. That is, look for consistent patterns in each row from the above figure.

Tons of manual work to reproduce what we knew!

Now that trusted the technology, we could get to work. We started visualizing gene activity patterns for genes that other scientists had previously identified as marker genes for the DLPFC layers. Some have identified in other organisms such as mouse 🐁. Some, due to the nature of Visium, we could not observe them. Visium can only measure 2,000 genes instead of the 20,000-30,000 genes that you can detect with other non-spatially-aware technologies.

Our paper describing our package #spatialLIBD is finally out! 🎉🎉🎉

spatialLIBD is an #rstats / @Bioconductor package to visualize spatial transcriptomics data.
⁰
This is especially exciting for me as it is my first paper as a first author 🦑.https://t.co/COW013x4GA

1/9 pic.twitter.com/xevIUg3IsA
— Brenda Pardo (@PardoBree) April 30, 2021

Data visualization, particularly with any new technology, is always a challenge. So we developed a new software that we called spatialLIBD to visualize the data (Pardo, Spangler, Weber, Hicks et al., 2022). In particular, we built spatialLIBD in a way that enabled members of our team to assign each Visium spot from our 12 tissue slices to one of seven categories: gray matter layers 1 to 6, and white matter. That was a lot of work! Many other people now refer to it as the “ground truth” although we know it is not. In total, we had 47,681 spots or an average of 3,973 spots per tissue slice. Imagine having to label that many fruits, you are bound to make some mistakes!

This was a terrific amount of work, and it is has been a fundamental reason why our study has been so well cited so far. Before I get ahead of myself, now that we have labeled all these spots, we can collapse the data and perform several types of analyses. Of them is looking at the different layers and white matter (WM) from the 12 tissue slices using principal component analysis (PCA). For a lot more on the mathematical theory of PCA check Wikipedia. What matters here is that PC1 explain the most differences, and we can see the WM points in black on the left side, clearly separated from the other points. Then PC2 is associated with L1 in pink at bottom of the Y-axis, to then L2 in blue, L3 in green, L4 in purple, L5 in yellow, and L6 in orange.

That was awesome! Yes, white matter is very different from gray matter (as shown in PC1), but then also the gray matter layers are different from each other in a sequential way (as shown in PC2). All of this was known before our study, but it always feels great when you can use a different technology to reproduce previous findings.

Finding and validating new DLPFC layer marker genes

Is that it? Of course not. Other technologies and laborious experiments were needed to find some marker genes for the DLPFC layers. But now we could identify other ones. So that’s what we did. Not only did we find genes that can help us separate these layers from each other, but we then proceeded to verify a few of them using other experiments.

The above figure showcases some of these new marker genes that have higher activity levels in the layer they mark. It doesn’t mean that they are not active elsewhere. Though if you are interested in finding on/off marker genes, we have computational methods for that too nowadays implemented in our DeconvoBuddies software.

We also propose and evaluate a new method for selecting cell type specific #markergenes for deconvolution: #MeanRatio implemented in #DeconvoBuddies (soon to be submited to @Bioconductor) #rstats
🛠️https://t.co/del9EjmBwQ pic.twitter.com/5PQk18yDrt
— Louise Huuki-Myers (@lahuuki) April 15, 2024

What about disorders? How does your `HumanPilot` study help us understand them?

As you have followed the story so far, our data is from 3 neurotypical control donors. It’s not like we can use their data (n = 3) to find what is overall different between neurotypical control donors and individuals affected by other brain disorders. But what we can do is ask, is there a significant overlap between the DLPFC layer marker genes and genes that have been associated with specific brain disorders?

So we obtained several gene lists from previous studies with much larger sample sizes that compared NTC donors against those affected by certain brain disorders. Some of these studies are based on gene activity levels, some of them are based on DNA variants which they then link to genes by proximity. One set of studies we looked at were those investigating autism spectrum disorder (ASD). What was fascinating to us is that two studies (SFARI and ASC102) had similar enrichment between ASD-associated genes and L2 and L5 marker genes. But the ASC102 study can be broken apart into two sets, ASD53 and DDID49, and each is associated with just one layer (L5 and L2, respectively).

So spatially-resolved transcriptomics could be used to further locate to specific tissue architectural elements the genes that have been implicated in different diseases, disorders, or biologically-relevant groups. Even small pilot studies like ours have the potential of helping us understand how some of these genes are distinctly spatially active.

Fueling the development of methods, and benefiting from it

We knew that this wasn’t going to be the end. Our HumanPilot was just the beginning and for our research groups, it effectively started a whole new wave of studies. We are now using Visium and other technologies to dive into other brain regions which aren’t necessarily as well studied as the DLPFC, or to look at differences between specific sets of groups of donors. Yet, this isn’t our first time witnessing the emergence of a brand new technology. We are well aware that each new technology has its imperfections, biases, and sources of measurement noise. One particular challenge we wanted help with was the process of manually annotating thousands of Visium spots.

In our dataset, we had manual annotations. So we went ahead and tested a few different ways of grouping spots, known as clustering.

Any computational clustering method will produce results and pretty pictures. By eye, sometimes you can tell quite easily what is a widely inaccurate result from one that seems decent enough. But comparing decent results from two methods by eye is very challenging.

We used a measure called adjusted rand index to do this, where higher values are better. Here we did a few key things:

We made our data public as soon as we pre-printed our study in Feb 29, 2020.
- That is, we didn’t wait for a full peer-review process (our paper was published in Feb 8th 2021, nearly a year later). Openly sharing data early can help accelerate science.
We made our data easy to access in a easy to use computational format.
- specifically, SingleCellExperiment R/Bioconductor objects downloadable through spatialLIBD::fetch_data(). Nowadays, we provide the data in the more specialized SpatialExperiment format.
We showed how you can compare results from clustering methods to the manual annotation using the adjusted rand index metric.

Within months, others such as the authors of BayesSpace in September 5th 2020, started to use our HumanPilot data to evaluate the performance of their sophisticated spatially-aware clustering methods.

This rapid pace of development has been great to witness. We now have been able to see how others have used our data to improve their methods, and we can now benefit from them by applying them to new Visium datasets without necessarily having to perform manual annotations.

What followed

Naturally we heard questions about what could happen if we had a larger pilot study. If we looked at different sections of DLPFC (anterior, middle, posterior). Today, our follow up peer-reviewed study was published involving data from 10 NTC donors across the anterior-posterior axis for a total of 30 Visium tissue slices. I’ll soon explain this follow up study, which we called spatialDLPFC, in its own blog post. Louise A. Huuki-Myers already wrote an excellent spatialDLPFC overview blog post which I encourage you to read.

New in @sciencemagazine: our work from @LieberInstitute #spatialDLPFC applies #snRNAseq and #Visium spatial transcriptomic in the DLPFC to better understand anatomical structure and cellular populations in the human brain #PsychENCODE https://t.co/DKZqmG4YDi https://t.co/Tjp2OjTo63 pic.twitter.com/vQbjts2JtQ
— Louise Huuki-Myers (@lahuuki) May 23, 2024

Something I want to highlight is that my colleague Kristen R. Maynard and I were co-first authors of HumanPilot, and are now co-corresponding authors of spatialDLPFC. HumanPilot started in 2019 and was published in 2021, spatialDLPFC started in 2020 and was published in 2024. These projects have large cycles and it takes a whole village, or a federation of villages, to get them completed and published. So I want to shout out all past colleagues. We couldn’t have gotten to where we are without their work! And thank you for everyone who has been interested in our work and has been using it to learn new things. We also learn through you. Thank you!!!

Self-assessment can be repetitive - or maybe not? 🤔 This caught my attention in recent spatial omics literature (but apparently not the authors'?)... "More of a comment than a question" that we need benchmarks. pic.twitter.com/KPRTf0vUxv
— Helena Lucia Crowell (@helucro) November 29, 2022

Hopefully you have a better understanding of our HumanPilot study now and can recognize the origin of all the figures like the ones above. If you want more, check out this journal club presentation by Cynthia Soto and Daianna Gonzalez-Padilla. They are members of my lab, but were not involved in this study at all. So their presentation has a fresh perspective.

Finally, feel free to checkout our code and more details at https://github.com/LieberInstitute/HumanPilot.

See you around! 😊

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2024)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[2\] K. R. Maynard, L. Collado-Torres, L. M. Weber, C. Uytingco, et al. “Transcriptome-scale spatial gene expression in the human dorsolateral prefrontal cortex”. In: Nature Neuroscience (2021). DOI: 10.1038/s41593-020-00787-0. URL: https://www.nature.com/articles/s41593-020-00787-0.

\[3\] A. Oleś. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.32.0. 2024. URL: https://github.com/Bioconductor/BiocStyle.

\[4\] B. Pardo, A. Spangler, L. M. Weber, S. C. Hicks, et al. “spatialLIBD: an R/Bioconductor package to visualize spatially-resolved transcriptomics data”. In: BMC Genomics (2022). DOI: 10.1186/s12864-022-08601-w. URL: https://doi.org/10.1186/s12864-022-08601-w.

\[5\] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

How to reduce the size of a large GitHub repo

Thu, 11 Apr 2024 00:00:00 +0000

This work was done by Leo with Erik Nelson and Ryan Miller. Thank you for your time and input!

Oh ohh! My GitHub repository is huge! 😱 What do I do now?

via GIPHY

Our story

That’s how things start. In our case, we had a repository that would download over 22.6 GB of data and use 55.9 GB of disk space. Note: This repository is currently (April 15 2024) private, but will be made public soon.

% git clone git@github.com:LieberInstitute/spatial_hpc.git
Cloning into 'spatial_hpc'...
remote: Enumerating objects: 18189, done.
remote: Counting objects: 100% (1999/1999), done.
remote: Compressing objects: 100% (1026/1026), done.
remote: Total 18189 (delta 963), reused 1981 (delta 950), pack-reused 16190
Receiving objects: 100% (18189/18189), 22.64 GiB | 12.90 MiB/s, done.
Resolving deltas: 100% (9734/9734), done.
Updating files: 100% (6667/6667), done.

We were able to reduce it such that it now downloads only 5.1 GB of data and takes 12.8 GB of disk space. While this could still be considered large by others, it’s much easier to download and tries to balance publicly providing some plots versus none.

% git clone git@github.com:LieberInstitute/spatial_hpc.git
Cloning into 'spatial_hpc'...
remote: Enumerating objects: 18451, done.
remote: Counting objects: 100% (8112/8112), done.
remote: Compressing objects: 100% (2851/2851), done.
remote: Total 18451 (delta 4129), reused 7963 (delta 4006), pack-reused 10339
Receiving objects: 100% (18451/18451), 5.13 GiB | 11.41 MiB/s, done.
Resolving deltas: 100% (9433/9433), done.
Updating files: 100% (5784/5784), done.

How were we able to achieve this? We used BFG Repo-Cleaner and removed all files over 10 MB except for two files that we wanted to keep.

Steps by steps

Get your workspace ready

We mostly work on a high performance computing (HPC) environment, aka a compute cluster, called JHPCE. In general for our projects, we have very large files that we do not version control. That is, we ignore them thanks to one or multiple .gitignore files. Given the complexity of differentiating version controlled files from ignored files, my recommendation is to work on your work computer (in my case, a laptop).

With this in mind, we did a fresh clone of the repository as shown previously.

% git clone git@github.com:LieberInstitute/spatial_hpc.git
Cloning into 'spatial_hpc'...
remote: Enumerating objects: 18189, done.
remote: Counting objects: 100% (1999/1999), done.
remote: Compressing objects: 100% (1026/1026), done.
remote: Total 18189 (delta 963), reused 1981 (delta 950), pack-reused 16190
Receiving objects: 100% (18189/18189), 22.64 GiB | 12.90 MiB/s, done.
Resolving deltas: 100% (9734/9734), done.
Updating files: 100% (6667/6667), done.

This is also the first step that BFG recommends that you do. In general, you want to make sure that you have a pristine copy that you can use as a fallback option in case anything goes wrong, and a working copy.

Note that we missed the --mirror option, although as you’ll see later, we did need to have a full copy of our files in our computer. So I actually recommend that you do not use --mirror.

We also downloaded BFG’s latest version: currently that is v1.14.0. This downloads the bfg-1.14.0.jar file. On a macOS computer, you also need to install Java from https://www.java.com/en/download/. As my macOS computer is from the M series system (ARM64), I did have to download the right version for it instead of the one from the big green button that downloads Java for macOS Intel (incompatible with my computer!).

You can then test that it’s all working as intended with this command:

% java -jar ~/Desktop/bfg-1.14.0.jar
bfg 1.14.0
Usage: bfg [options] [<repo>]

  -b, --strip-blobs-bigger-than <size>
                           strip blobs bigger than X (eg '128K', '1M', etc)
  -B, --strip-biggest-blobs NUM
                           strip the top NUM biggest blobs
  -bi, --strip-blobs-with-ids <blob-ids-file>
                           strip blobs with the specified Git object ids
  -D, --delete-files <glob>
                           delete files with the specified names (eg '*.class', '*.{txt,log}' - matches on file name, not path within repo)
  --delete-folders <glob>  delete folders with the specified names (eg '.svn', '*-tmp' - matches on folder name, not path within repo)
  --convert-to-git-lfs <value>
                           extract files with the specified names (eg '*.zip' or '*.mp4') into Git LFS
  -rt, --replace-text <expressions-file>
                           filter content of files, replacing matched text. Match expressions should be listed in the file, one expression per line - by default, each expression is treated as a literal, but 'regex:' & 'glob:' prefixes are supported, with '==>' to specify a replacement string other than the default of '***REMOVED***'.
  -fi, --filter-content-including <glob>
                           do file-content filtering on files that match the specified expression (eg '*.{txt,properties}')
  -fe, --filter-content-excluding <glob>
                           don't do file-content filtering on files that match the specified expression (eg '*.{xml,pdf}')
  -fs, --filter-content-size-threshold <size>
                           only do file-content filtering on files smaller than <size> (default is 1048576 bytes)
  -p, --protect-blobs-from <refs>
                           protect blobs that appear in the most recent versions of the specified refs (default is 'HEAD')
  --no-blob-protection     allow the BFG to modify even your *latest* commit. Not recommended: you should have already ensured your latest commit is clean.
  --private                treat this repo-rewrite as removing private data (for example: omit old commit ids from commit messages)
  --massive-non-file-objects-sized-up-to <size>
                           increase memory usage to handle over-size Commits, Tags, and Trees that are up to X in size (eg '10M')
  <repo>                   file path for Git repository to clean

Diagnose

Once you have BFG installed as well as a working clone (copy) of your GitHub repository, you can proceed to identify large files. Say files larger than 100 MB. That is the strategy that you typically use when you version controlled large files and made other smaller commits, and now cannot git push (upload files) to GitHub. In our case, we had no files over 100 MB.

% cd ~/Desktop/spatial_hpc
% java -jar ~/Desktop/bfg-1.14.0.jar --strip-blobs-bigger-than '100M' .git

Using repo : /Users/leocollado/Dropbox/Mac/Desktop/spatial_hpc/.git

Scanning packfile for large blobs: 18215
Scanning packfile for large blobs completed in 89 ms.
Warning : no large blobs matching criteria found in packfiles - does the repo need to be packed?

So we then proceeded to check files over 10 MB (we also checked files over 50 MB but we won’t talk about that analysis in this blog post). BFG provides a lot of output as shown below. BFG told us that we had 885 “dirty files”. These are files over 10 MB (the threshold value we specified) that are present in our latest version on GitHub. It also found “clean files” over 10 MB, which are files that at some point we had version controlled but then deleted them. So they only exist in the git history in case we need to go back in time, but are not visible from the landing page on GitHub (though you can use GitHub to navigate the git history if you want to).

% java -jar ~/Desktop/bfg-1.14.0.jar --strip-blobs-bigger-than '10M' .git

Using repo : /Users/leocollado/Dropbox/Mac/Desktop/spatial_hpc/.git

Scanning packfile for large blobs: 18215
Scanning packfile for large blobs completed in 91 ms.
Found 1130 blob ids for large blobs - biggest=97378037 smallest=10491618
Total size (unpacked)=29298335089
Found 6108 objects to protect
Found 4 commit-pointing refs : HEAD, refs/heads/main, refs/remotes/origin/HEAD, refs/remotes/origin/main

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit f5f9a19e (protected by 'HEAD') - contains 885 dirty files :
	- code/enrichment_analysis/project_all/score/snRNAseq_aggregated_cpm.tsv (16.4 MB)
	- code/spot_deconvo/cell2location/02_registration_HE_broad_class.log (16.3 MB)
	- ...

WARNING: The dirty content above may be removed from other commits, but as
the *protected* commits still use it, it will STILL exist in your repository.

Details of protected dirty content have been recorded here :

/Users/leocollado/Dropbox/Mac/Desktop/spatial_hpc/.git.bfg-report/2024-04-11/09-51-45/protected-dirt/

If you *really* want this content gone, make a manual commit that removes it,
and then run the BFG on a fresh copy of your repo.


Cleaning
--------

Found 1561 commits
Cleaning commits:       100% (1561/1561)
Cleaning commits completed in 1,197 ms.

Updating 2 Refs
---------------

	Ref                        Before     After
	----------------------------------------------
	refs/heads/main          | f5f9a19e | ba48bc2c
	refs/remotes/origin/main | f5f9a19e | ba48bc2c

Updating references:    100% (2/2)
...Ref update completed in 12 ms.

Commit Tree-Dirt History
------------------------

	Earliest                                              Latest
	|                                                          |
	..DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD

	D = dirty commits (file tree fixed)
	m = modified commits (commit message or parents changed)
	. = clean commits (no changes to file tree)

	                        Before     After
	-------------------------------------------
	First modified commit | 545cd363 | fcc2538e
	Last dirty commit     | ead4cbed | 600580dd

Deleted files
-------------

	Filename                                                          Git id
	----------------------------------------------------------------------------------------------------------------------------
	02_registration_HE_broad_class.log                              | de11acf2 (12.9 MB), 33ae1f4d (16.3 MB)
	02_registration_HE_layer.log                                    | ba6d1c7e (19.6 MB), 279e5354 (19.6 MB)
	2743.png                                                        | c734cb7d (11.3 MB)
	335.png                                                         | 602fc07e (18.8 MB)
	6423.png                                                        | 7071add9 (15.4 MB)
	6432.png                                                        | 9b9d7fa6 (13.2 MB)
	6471.png                                                        | 0f61bfd7 (17.5 MB)
	6522.png                                                        | 573fab2b (19.8 MB)
	8325.png                                                        | 9cf4ff3e (22.0 MB)
	8492.png                                                        | 0a128ff1 (10.2 MB)
	8667.png                                                        | 014ee77a (15.4 MB)
	AC005837.4.pdf                                                  | 0b12e443 (32.4 MB)
	AC106795.2.pdf                                                  | 60dba623 (32.4 MB), ba037e7f (32.5 MB)
	AC108863.2.pdf                                                  | 56eb980a (32.4 MB)
	ACAP1_DOWN.pdf                                                  | 584afb1f (32.5 MB)
	...


In total, 5160 object ids were changed. Full details are logged here:

	/Users/leocollado/Dropbox/Mac/Desktop/spatial_hpc/.git.bfg-report/2024-04-11/09-51-45

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive

Among the output of BFG was a CSV file which listed the 885 “dirty files”.

% ls -lh /Users/leocollado/Dropbox/Mac/Desktop/spatial_hpc/.git.bfg-report/2024-04-11/09-51-45/protected-dirt/
total 132K
-rw-r--r-- 1 leocollado staff 132K Apr 11 09:51 f5f9a19e-HEAD.csv

With R, we can read that CSV file and help us diagnose our situation. We recommend that you inspect this CSV file, which will be ordered by file path.

## Read in the CSV file with R
ten <- read.csv("f5f9a19e-HEAD.csv", header = FALSE)

## How many "dirty files" do we have?
nrow(ten)

## [1] 885

## What's the format of this CSV file?
head(ten)

##                                         V1     V2           V3
## 1 e00fd6349fcabdd740b396b230a74f9f0d136f2e DELETE   executable
## 2 33ae1f4da994efaad1538c6d740ac22fb6f0aab4 DELETE regular-file
## 3 279e53549e75b881fb7b063f05211fde529c17fb DELETE regular-file
## 4 8e1c82f0c40758394abfc807a3fcc6094fef329d DELETE regular-file
## 5 f41fa6e0e3e6a25b390aa33a4d4fa3c3b493d97a DELETE regular-file
## 6 89f9bb0104d49d7584d712a8cde2e9a580356291 DELETE regular-file
##                                                                       V4
## 1 code/enrichment_analysis/project_all/score/snRNAseq_aggregated_cpm.tsv
## 2     code/spot_deconvo/cell2location/02_registration_HE_broad_class.log
## 3           code/spot_deconvo/cell2location/02_registration_HE_layer.log
## 4                                plots/02_build_spe/ReferenceMapping.pdf
## 5                           plots/02_build_spe/ReferenceMapping_erik.pdf
## 6                           plots/02_build_spe/prearrangedAllSamples.pdf
##         V5 V6
## 1 17229010 NA
## 2 17115444 NA
## 3 20518064 NA
## 4 54308266 NA
## 5 54308266 NA
## 6 25909369 NA

The fourth column specifies the file path and the fifth column specifies the file size in bytes. Given that this project was organized according to our template_project, Erik (who has been heavily involved in this project) could easily diagnose whether some of the listed files were important to share or whether they were older exploration results.

## Sort them by file size if you want to
# ten <- ten[order(ten$V5, decreasing = TRUE), ]

## Check file size distribution in MB (1 MB = 1024^2 bytes)
summary(ten$V5 / 1024^2)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   10.01   19.08   26.12   26.31   32.50   92.87

boxplot(ten$V5 / 1024^2)

## Explore frequency of file sizes rounded to 1 digit
barplot(table(round(ten$V5 / 1024^2, 1)))

## How much total space do these "dirty files" use?
sum(ten$V5 / 1024^2) ## In MB

## [1] 23285.45

sum(ten$V5 / 1024^3) ## In GB

## [1] 22.73969

A question Erik posed was: is it mostly plots? In general, you could use file extensions if you wanted. But in our case, thanks to template_project, we can check what is the main directory for each file.

## Extract main directory from the file paths
x <- dirname(ten$V4)
x <- gsub("/.*", "", x)

## Explore how many files we have
table(x)

## x
##           code          plots processed-data       raw-data   snRNAseq_hpc 
##              3            505            368              2              7

We could then explore some of the odd cases.

## Visual inspection determined that we were OK deleting these files
ten[x == "snRNAseq_hpc", ]

##                                           V1     V2           V3
## 879 f0d8bb12bbadda84ae3e98f431083e61620869b2 DELETE   executable
## 880 68b61e75d861f5d960a96cbdcad7bb5c79a3e747 DELETE regular-file
## 881 f7588d013685ebe907c972afd0aaf0935816df28 DELETE regular-file
## 882 771e792004afef8ed2cc986eb9766b01bad4db4a DELETE regular-file
## 883 204d559465dba1489955d5988a879c6cbb91a5b7 DELETE regular-file
## 884 f5bb06176473e8112d00a1d355ccd048f4dc46c1 DELETE regular-file
## 885 605957a9d3a9b2f2ed8d021dcf2c1482048f4f9a DELETE regular-file
##                                                               V4       V5 V6
## 879                        snRNAseq_hpc/plots/02_qc/qc_plots.pdf 20223555 NA
## 880                               snRNAseq_hpc/plots/QC_auto.pdf 41669358 NA
## 881                           snRNAseq_hpc/plots/QC_semiauto.pdf 16580972 NA
## 882                          snRNAseq_hpc/plots/QC_semiauto2.pdf 33358656 NA
## 883    snRNAseq_hpc/plots/build_sce/UMAP_corrected_by_sample.pdf 18167086 NA
## 884          snRNAseq_hpc/plots/build_sce/UMAPs_unccorrected.pdf 18047630 NA
## 885 snRNAseq_hpc/plots/build_sce/UMAPs_unccorrected_filtered.pdf 16767377 NA

## Same thing for these large "code" files (2 logs and one file that is not
## needed to run the actual code and thus reproduce results)
ten[x == "code", ]

##                                         V1     V2           V3
## 1 e00fd6349fcabdd740b396b230a74f9f0d136f2e DELETE   executable
## 2 33ae1f4da994efaad1538c6d740ac22fb6f0aab4 DELETE regular-file
## 3 279e53549e75b881fb7b063f05211fde529c17fb DELETE regular-file
##                                                                       V4
## 1 code/enrichment_analysis/project_all/score/snRNAseq_aggregated_cpm.tsv
## 2     code/spot_deconvo/cell2location/02_registration_HE_broad_class.log
## 3           code/spot_deconvo/cell2location/02_registration_HE_layer.log
##         V5 V6
## 1 17229010 NA
## 2 17115444 NA
## 3 20518064 NA

## These are important sample information files we do want to keep.
ten[x == "raw-data", ]

##                                           V1     V2           V3
## 877 f44d0d43bd507c0e33e9267ae1ba00fda71acf66 DELETE regular-file
## 878 a3dd0e6f6056de3c11b4f3804b5ee40b39eabef4 DELETE regular-file
##                                                                                        V4
## 877 raw-data/FASTQ/2022-12-07_NovaSeq_33v_scp/Visium_HPC_Round1-10_092622_Master_SCP.xlsx
## 878                              raw-data/sample_info/Visium_HPC_Round3-8_033022_SCP.xlsx
##           V5 V6
## 877 37226953 NA
## 878 34649463 NA

Remove dirty files

Once we have identified which files we no longer want to distribute through GitHub, we can proceed to eliminate them from the git history in our local working copy. Before we can use BFG, we first have to clean these “dirty files”. The first step is to remove them from the list of version controlled files using the git rm command. However, we do want to keep a copy of these files. So we use the --cached option to do so.

## Change directory to the location of your local copy
setwd("~/Desktop/spatial_hpc/")

## Create a list of commands
cmds <- paste0("git rm --cached ", ten[x != "raw-data",]$V4)

## Loop over and execute the commands with system()
for (i in seq_along(cmds)) {
    message(i, ": ", Sys.time(), " ", cmds[i])
    system(cmds[i])
}

Here’s how one of these commands would look like. We used system() to actually run these commands on my macOS computer.

cmds[1]

## [1] "git rm --cached code/enrichment_analysis/project_all/score/snRNAseq_aggregated_cpm.tsv"

At this point, with git rm we have told git to no longer version control these 880+ files. To avoid the mistake of version controlling them again, we then added them to the main .gitignore file using readLines() and writeLines().

## Update our main .gitignore file with the list of files
## we no longer want to version control.
gitignore <- readLines(".gitignore")
gitignore <- c(gitignore, ten[x != "raw-data",]$V4)
writeLines(gitignore, ".gitignore")

At this point, if you use git status, you’ll see the long list of files marked for removal from version control.

% git status
On branch main
Your branch is ahead of 'origin/main' by 1 commit.
  (use "git push" to publish your local commits)

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	deleted:    plots/02_build_spe/ReferenceMapping.pdf
	deleted:    plots/02_build_spe/ReferenceMapping_erik.pdf
	deleted:    plots/02_build_spe/prearrangedAllSamples.pdf
	
## and many more lines

## Note that you should not see any untracked files listed at this step.

Next, you have to make a git commit to finally unprotect these “dirty files”.

git commit -am "Unprotecting files over 10 Mb so we can remove them from git history with BFG"

Editing the `git` history

Now that we no longer have “dirty files” over 10 MB (except for the 2 Excel files we decided to keep), we can now run BFG again. At this point, out of precaution, I did make a full copy of my local clone (copy) of the repository. Though in retrospect, you likely want to do this before you even run BFG the very first time. That’s because BFG will detect that it has been used before on a given git repository and will prune repo before proceeding which took about 6 minutes for us.

% date
Thu Apr 11 11:27:06 EDT 2024

% java -jar ~/Desktop/bfg-1.14.0.jar --strip-blobs-bigger-than '10M' .git

Using repo : /Users/leocollado/Dropbox/Mac/Desktop/spatial_hpc/.git


This repo has been processed by The BFG before! Will prune repo before proceeding - to avoid unnecessary cleaning work on unused objects...


date
Completed prune of old objects - will now proceed with the main job!

Scanning packfile for large blobs: 42927
Scanning packfile for large blobs completed in 672 ms.
Found 1130 blob ids for large blobs - biggest=97378037 smallest=10491618
Total size (unpacked)=29298335089
Found 5225 objects to protect
Found 4 commit-pointing refs : HEAD, refs/heads/main, refs/remotes/origin/HEAD, refs/remotes/origin/main

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 3d58fd38 (protected by 'HEAD') - contains 2 dirty files :
	- raw-data/FASTQ/2022-12-07_NovaSeq_33v_scp/Visium_HPC_Round1-10_092622_Master_SCP.xlsx (35.5 MB)
	- raw-data/sample_info/Visium_HPC_Round3-8_033022_SCP.xlsx (33.0 MB)

WARNING: The dirty content above may be removed from other commits, but as
the *protected* commits still use it, it will STILL exist in your repository.

Details of protected dirty content have been recorded here :

/Users/leocollado/Dropbox/Mac/Desktop/spatial_hpc/.git.bfg-report/2024-04-11/11-33-47/protected-dirt/

If you *really* want this content gone, make a manual commit that removes it,
and then run the BFG on a fresh copy of your repo.


Cleaning
--------

Found 1563 commits
Cleaning commits:       100% (1563/1563)
Cleaning commits completed in 1,390 ms.

Updating 1 Ref
--------------

	Ref               Before     After
	-------------------------------------
	refs/heads/main | 3d58fd38 | d598aefb

Updating references:    100% (1/1)
...Ref update completed in 12 ms.

Commit Tree-Dirt History
------------------------

	Earliest                                              Latest
	|                                                          |
	...........................................................D

	D = dirty commits (file tree fixed)
	m = modified commits (commit message or parents changed)
	. = clean commits (no changes to file tree)

	                        Before     After
	-------------------------------------------
	First modified commit | 0255622c | 35eacffa
	Last dirty commit     | 0255622c | 35eacffa

Deleted files
-------------

	Filename                                                          Git id
	----------------------------------------------------------------------------------------------------------------------------
	02_registration_HE_broad_class.log                              | de11acf2 (12.9 MB), 33ae1f4d (16.3 MB)
	02_registration_HE_layer.log                                    | ba6d1c7e (19.6 MB), 279e5354 (19.6 MB)
	2743.png                                                        | c734cb7d (11.3 MB)
	335.png                                                         | 602fc07e (18.8 MB)
	6423.png                                                        | 7071add9 (15.4 MB)
	6432.png                                                        | 9b9d7fa6 (13.2 MB)
	6471.png                                                        | 0f61bfd7 (17.5 MB)
	6522.png                                                        | 573fab2b (19.8 MB)
	8325.png                                                        | 9cf4ff3e (22.0 MB)
	8492.png                                                        | 0a128ff1 (10.2 MB)
	8667.png                                                        | 014ee77a (15.4 MB)
	AC005837.4.pdf                                                  | 0b12e443 (32.4 MB)
	AC106795.2.pdf                                                  | 60dba623 (32.4 MB), ba037e7f (32.5 MB)
	AC108863.2.pdf                                                  | 56eb980a (32.4 MB)
	ACAP1_DOWN.pdf                                                  | 584afb1f (32.5 MB)
	...


In total, 5168 object ids were changed. Full details are logged here:

	/Users/leocollado/Dropbox/Mac/Desktop/spatial_hpc/.git.bfg-report/2024-04-11/11-33-47

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive

% date
Thu Apr 11 11:33:49 EDT 2024

As we can see on the output above, we do have 2 “dirty files”, but we knew about them. We can now proceed to run the commands BFG tells us to use to actually edit the git history.

git reflog expire --expire=now --all && git gc --prune=now --aggressive

We went to get lunch while this was happening, as my recollection was that this step can take a while to run. Though I forgot to time it this time around.

Update `GitHub`

We now have what we want: a much smaller git repository with only the files we want version controlled. However, it’s not on GitHub and also not in our HPC computers (JHPCE for us). So we need to proceed to update GitHub first using git push (upload command). As we are altering the git history, we need to use a git push --force command otherwise GitHub would not let us do alter the history.

% git push --force
Enumerating objects: 18451, done.
Counting objects: 100% (18451/18451), done.
Delta compression using up to 10 threads
Compressing objects: 100% (6684/6684), done.
remote: fatal: pack exceeds maximum allowed size (2.00 GiB)
error: remote unpack failed: index-pack failed 14.02 MiB/s
To github.com:LieberInstitute/spatial_hpc.git
 ! [remote rejected]   main -> main (failed)
error: failed to push some refs to 'github.com:LieberInstitute/spatial_hpc.git'

However, as you can see, we ran into the 2 GB limit GitHub has for git push commands.

Google searching the error, we easily found this excellent documentation page. It didn’t exist (or I didn’t find it) a few years ago and it was quite straightforward to understand in contrast to previous Stack Overflow answers I remember struggling with back then.

Looking at our repository on GitHub, we saw that we have a bit over 1,500 commits. So searching every 1,000 commits most likely wasn’t going to work for us as the GitHub documentation suggested given that we would most likely run again into the 2 GB upload limit. We decided to roll with 200 commits at a time hoping that this would give us a small enough set of commit IDs to make incremental git pushes, but without exceeding the 2 GB upload limit GitHub enforces.

% git log --oneline --reverse refs/heads/main | awk 'NR % 200 == 0'
d983b009 marker gene plots
df46ec6e log file for BayesSpace rerun k15, long array
066777d0 Added a README.md file and softlinks for round9 (samples 33v-36v)[33v sample (10ul) library MiSeq and NovaSeq,samples 34v-26v NovaSeq]
51ec749f rerun VNS for sample 36 with more memory
fd032a96 tangram logs 4,5,6
e9e6526f run updated code on sample 24
9572195c updated samui image stitching script to remove banding from center of the image and average pixel intensity of the image overlaps

Thanks to the commit messages, Erik and Ryan could easily detect that the commit IDs were ordered by time such that the last commit was one of the most recent ones. We thus proceeded to run each of the git push commands. Note that we used git push --force on our first commit. After this, our repository on GitHub listed only 200 commits: what we wanted to do.

% git push --force origin d983b009:refs/heads/main
Enumerating objects: 2253, done.
Counting objects: 100% (2253/2253), done.
Delta compression using up to 10 threads
Compressing objects: 100% (1009/1009), done.
Writing objects: 100% (2253/2253), 633.78 MiB | 36.74 MiB/s, done.
Total 2253 (delta 1176), reused 1987 (delta 1085), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (1176/1176), done.
To github.com:LieberInstitute/spatial_hpc.git
 + f5f9a19e...d983b009 d983b009 -> main (forced update)

Some of them got close to the 2 GB limit (like 1.5 GB), but they were all fine.

## Commands shown with no output
% git push origin df46ec6e:refs/heads/main
% git push origin 066777d0:refs/heads/main
% git push origin 51ec749f:refs/heads/main
% git push origin fd032a96:refs/heads/main
% git push origin e9e6526f:refs/heads/main
% git push origin 9572195c:refs/heads/main
% git push

Note the last git push we did at the end, which was needed to upload the last left over commits. Also note that this step was fairly simple for us as we only have 1 development branch called main for this project, instead of working across multiple git branches. If you are working with multiple branches, you might have to do quite a bit of extra work.

Updating HPC files

Are we done yet? Close! We have what we want in our local git clone (copy) on my laptop as well as at GitHub, but not in our HPC computers (JHPCE for us). We followed this Stack Overflow solution to update files at JHPCE:

## Log in to your HPC system (JHPCE for us)

## Navigate to the directory where the data lives at JHPCE
cd /dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc

## Download all files from GitHub to our HPC (JHPCE)
git fetch

## Force update the "main" branch
git reset origin/main --hard

At this point, we did run into a warning because I didn’t have the permissions to delete one file.

warning: unable to unlink 'code/enrichment_analysis/project_all/score/snRNAseq_aggregated_cpm.tsv': Permission denied

As you might already be wondering, git reset origin/main --hard did erase all 880+ files (except the one I didn’t have permissions for) over 10 Mb that we deleted from git and GitHub. But we wanted to have them at JHPCE. This is where having a local copy in our laptop becomes handy. We then used rsync to upload back these files to our HPC computers from my laptop.

## From your computer (my laptop in this case), upload the local files you have
## that were deleted from git / GitHub that you want to have in your HPC system
rsync -avhP . lcollado@jhpce-transfer01.jhsph.edu:/dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc

However, this process broke some file permissions. As we already have a script for updating permissions, I simply re-ran it at JHPCE with:

## Log in to your HPC system (JHPCE for us)

## Navigate to the directory where the data lives at JHPCE
$ cd /dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc

## Run script for updating permissions
$ sh /dcs04/lieber/lcolladotor/_jhpce_org_LIBD001/update_permissions_spatialteam.sh .
**** Updating permissions for . ****
Thu Apr 11 01:49:01 PM EDT 2024

**** Note that warning/error messages are expected for files and directories that you are not the owner of.
The expected warning/error messages are:
    'chgrp: changing group of ‘some_JHPCE_file_path’: Operation not permitted'
 or 'chmod: changing permissions of ‘some_JHPCE_file_path’: Operation not permitted'.
If for many files you are not the owner (creator of), you will get lots of these warning/error messages, this is expected!
Error or warnings with another syntax are likely real. ****

You will need to re-run this script anytime you upload files to JHPCE through Cyberduck / WinSCP as they break the ACLs.
Every new team member on a given project will likely also need to run this script once.

For more details about setting permissions at JHPCE using ACLs, please check https://lcolladotor.github.io/bioc_team_ds/organizing-your-work.html#setting-jhpce-file-permissions.

This message will be displayed for 90 seconds before the script proceeds.
That way you will have time enough to read it and/or copy it.

**** Setting read (R), write (W), and execute (X) permissions for hickslab ****
Thu Apr 11 01:50:36 PM EDT 2024


## truncated output

**** Setting the group sticky bit ****
Thu Apr 11 03:05:53 PM EDT 2024

**** Checking the nfs4 (ACLs) settings ****
Thu Apr 11 03:07:11 PM EDT 2024
# file: .
A::OWNER@:rwaDxtTcCy
A::GROUP@:rxtcy
A:g:lieber_lcolladotor@cm.cluster:rwaDxtcy
A:g:lieber_marmaypag@cm.cluster:rwaDxtcy
A:g:hickslab@cm.cluster:rxtcy
A::EVERYONE@:rxtcy
A:fdi:OWNER@:rwaDxtTcCy
A:fdi:GROUP@:rwaDxtcy
A:fdig:lieber_lcolladotor@cm.cluster:rwaDxtcy
A:fdig:lieber_marmaypag@cm.cluster:rwaDxtcy
A:fdig:hickslab@cm.cluster:rwaDxtcy
A:fdi:EVERYONE@:tcy

Now everyone else involved in this team project can keep working with the files at JHPCE and GitHub.

Finishing step

Erik will check if any of the 880+ files over 10 MB that we deleted from the git history (and currently have at JHPCE but are not on GitHub ) are actually files we should have on GitHub. If so, he’ll edit the main .gitignore accordingly (remove them from the main .gitignore ) + then git add them again.

Summary

This whole process took about three and a half hours. I hadn’t done something like this in a while so my memory wasn’t fresh. That’s why I’m writing this blog post, so we can have a good guide for the next time we have to do this.

Here are some main notes:

Announce about a week in advance that no one should be working on the files at your HPC system (JHPCE for us): “closed for maintenance ⚠️”
- This will give people enough time to plan ahead.
Ask people to run the update permissions script in advance.
- We got lucky this time with encountering only 1 restricted file!
Work on a fresh git clone in your laptop
Feel free to make a copy on your laptop before you use BFG
Work with people who are very familiar with the contents of the repository.
- They will make crucial decisions on which files to keep or not!!
Allocate plenty of time.
- We spent about 4 hours, but you might want to plan for 4-6 hours.
Don’t copy paste more than 1 command into the terminal at a time.
- You might encounter errors not previously documented!

Enjoy! Best of luck with navigating similar situations.

via GIPHY

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2024)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

[1] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

[2] A. Oleś. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.32.0. 2024. URL: https://github.com/Bioconductor/BiocStyle.

[3] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.4.0 (2024-04-24)
##  os       macOS Sonoma 14.5
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2024-05-23
##  pandoc   3.2 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.4.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.4.0)
##  BiocManager     1.30.23    2024-05-04 [1] CRAN (R 4.4.0)
##  BiocStyle     * 2.32.0     2024-04-30 [1] Bioconductor 3.19 (R 4.4.0)
##  blogdown        1.19       2024-02-01 [1] CRAN (R 4.4.0)
##  bookdown        0.39       2024-04-15 [1] CRAN (R 4.4.0)
##  bslib           0.7.0      2024-03-29 [1] CRAN (R 4.4.0)
##  cachem          1.1.0      2024-05-16 [1] CRAN (R 4.4.0)
##  cli             3.6.2      2023-12-11 [1] CRAN (R 4.4.0)
##  colorout      * 1.3-0.2    2024-05-03 [1] Github (jalvesaq/colorout@c6113a2)
##  digest          0.6.35     2024-03-11 [1] CRAN (R 4.4.0)
##  evaluate        0.23       2023-11-01 [1] CRAN (R 4.4.0)
##  fastmap         1.2.0      2024-05-15 [1] CRAN (R 4.4.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.4.0)
##  glue            1.7.0      2024-01-09 [1] CRAN (R 4.4.0)
##  highr           0.10       2022-12-22 [1] CRAN (R 4.4.0)
##  htmltools       0.5.8.1    2024-04-04 [1] CRAN (R 4.4.0)
##  httr            1.4.7      2023-08-15 [1] CRAN (R 4.4.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.4.0)
##  jsonlite        1.8.8      2023-12-04 [1] CRAN (R 4.4.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.4.0)
##  knitr           1.46       2024-04-06 [1] CRAN (R 4.4.0)
##  lifecycle       1.0.4      2023-11-07 [1] CRAN (R 4.4.0)
##  lubridate       1.9.3      2023-09-27 [1] CRAN (R 4.4.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.4.0)
##  plyr            1.8.9      2023-10-02 [1] CRAN (R 4.4.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.4.0)
##  Rcpp            1.0.12     2024-01-09 [1] CRAN (R 4.4.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.4.0)
##  rlang           1.1.3      2024-01-10 [1] CRAN (R 4.4.0)
##  rmarkdown       2.27       2024-05-17 [1] CRAN (R 4.4.0)
##  rstudioapi      0.16.0     2024-03-24 [1] CRAN (R 4.4.0)
##  sass            0.4.9.9000 2024-05-03 [1] Github (rstudio/sass@9228fcf)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.4.0)
##  stringi         1.8.4      2024-05-06 [1] CRAN (R 4.4.0)
##  stringr         1.5.1      2023-11-14 [1] CRAN (R 4.4.0)
##  timechange      0.3.0      2024-01-18 [1] CRAN (R 4.4.0)
##  xfun            0.44       2024-05-15 [1] CRAN (R 4.4.0)
##  xml2            1.3.6      2023-12-04 [1] CRAN (R 4.4.0)
##  yaml            2.3.8      2023-12-11 [1] CRAN (R 4.4.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

CZI Lat Am Meeting 2023: some questions I have before the meeting

Sun, 16 Apr 2023 00:00:00 +0000

I’m excited for the opportunity to participate at the #CZILatAmMtg in 2023 organized by CZI Science. I’m also thankful to those who nominated me, and in particular to Alejandra Medina Rivera. I believe that one of the reasons I’m here is for my volunteer work at the Community of Bioinformatics Software Developers in Mexico (CDSB), where many have made very important contributions. Some also know me from my time in the Bioconductor Community Advisory Board, from which I stepped down earlier this year once my 3 year term ended.

Having said that, I have been sharing with some other event attendees that I do feel burned out. I had envisioned that at CDSB we would train and empower those who would eventually replace us. And we’ve had quite a bit of success in doing so (3/7 CDSB board members are not founders). However, it’s been limited to a few key new volunteers, who then they risk burning out themselves. Since I want CDSB to continue on for many more years to come, I sometimes feel like I should again involve myself on organizing CDSB events. Doing so would involve volunteering on my free time. Given my position as a principal investigator, I actually spend more extra hours on activities related to that job, so I don’t have much free time to give. There are also other non-academic interests I want to pursue.

Naturally, I want to chat with others that organize short courses through their own organizations to learn how they plan to pass on the torch and see if those ideas can apply to CDSB as well as renew my own fire. Maybe others don’t plan to pass on the torch for many years as organizing these short courses becomes part of their main job. At CDSB (and LCG-UNAM) courses, I like the path of having students in year 1, that become teaching assistants in year 2, that then teach a portion of the course in year 3, then can organize part of or a full course in year 4. But that’s a long ramp, and I think that I failed to anticipate what would happen after year 4. Like there needs to be more structure to help them continue this cycle and reach a point where they can apply for grants without the names of the founders. I also failed to notice that founders would need to be involved for a longer period of time, until we reach a larger critical mass of highly trained volunteers, which at time point can fully replace us and keep CDSB going for more years to come.

However, I also have other questions and topics I want to discuss with others. For example, I’m really curious to learn about justifications others use to convince their employers to let them spend work time on these type of initiatives.

Some arguments I’ve used are:

We are training the future staff we want to recruit. I learned this from a former SACNAS president and it was useful to argue for time to teach (in my case in Mexico) without having to take vacations time to do so for a week or two per year.
This is one I used recently in the context of the LIBD rstats club I organize at work: “YouTube analytics tell me that we’ve had 4,621 views in the last 365 days for a total watch time of 280.7 hours. That’s an average of 5.4 hours per week, which is 13.5% of a 40 hour work week that we saved a full time employee on average per week this last year. For example, for a base salary of $x$ or $y$ that’s >10,000 or >20,000 USD in savings per year taking into account fringe costs. These videos remain in perpetuity, and while some might become outdated, they are definitely an asset for the institute.”

I’m also curious to hear what arguments others use to convince their employers to let them apply to grants that limit indirect costs. For example, US-based institutions depend on those indirect costs to function, and will not like it if you apply for a grant that will limit your effort available for other grants if that grant provides, let’s say, less than 25% or 50% of the indirect costs other grants provide. Direct costs are like research costs and salary support for your team. Like if you are doing so well that you get funding for 100% of your effort, then of course institutions would like it for you to have the max indirect costs rate possible. Though well, a grant with a lower indirect costs rate is better than no grant if you don’t have 100% funded effort.

Maybe something funding bodies could do is request that a small portion of the grantees time is used to teach communities in need for these opportunities (here we are talking about Latin America, but it could be other ones). Another idea is that they could reward with a higher indirect cost rate those institutions that allow grantees to do this during work hours. These ideas might have no legs, but maybe they could. We’ll see.

While change takes a lot of effort and sometimes one might feel alone, I am definitely excited about the discussions we’ll have the next two days. Though I’m also aware of potential pitfalls, like fighting for a piece of the cake. Hopefully we can make the cake bigger!

via GIPHY

I'm disappointed by the reaction against the Tokyo 2020 Mexican Women Softball team

Fri, 30 Jul 2021 00:00:00 +0000

Yesterday there was a controversy because the Mexican Women’s softball team at the Tokyo 2020 Olympics (Wikipedia, Instagram) threw several uniforms and equipment to the trash (source: ESPN). Right now I’m really disappointed with the negative reaction and personal attacks against the members of said team. At the same time, I don’t want to remain silent so I decided to write this post.

Imagine that you live outside of your ancestral country. People see you differently and tell you to go back to the country you came from. This is a reality that inmigrants around the world live through frequently, including Mexicans and their descendants in the US. If this happens to you, it can generate a strong internal reaction. Descendents from inmigrants have to learn to live with and adjust to their multiple identities. In my experience it’s a gradient and I see it with friends that just arrived to the US from Mexico, people like myself who have been in the US for years, those who inmigrated to the US as children, and those who are descendants (2nd generation and beyond). Everyone is different and each one of us identifies with their cultural roots in different ways.

View this post on Instagram

A post shared by Anissa Urtez (@nniiiss)

That’s why the experience of Anissa Urtez, one of the members of this team ^[Why Anissa Urtez? She’s the one that shared the most detailed information of what happened through her Instagram stories. Furthermore, Anissa was selected to the ideal women’s softball team for the Tokyo 2020 Olympics.], that she describes of growing as an American with Mexican ancestors where she wasn’t considered Mexican enough, American enough, and with brown skin, is totally valid and complicated. If six years ago she decided to be part of a low resources project that aimed to build a Mexican Women’s softball team that could celeberate winning a match, participate in tournaments, win enough in the 2019 Pan Aamerican Games to classify to the Olympics for the first time ever, we have to thank her and value her effort, time, energy, and all that she dedicated towards making this dream a reality. Her participation at the Tokyo 2020 Olympics was never guaranteed.

View this post on Instagram

A post shared by Anissa Urtez (@nniiiss)

And they made this dream a reality. As a mexican I’m overwhelmed with emotion and pride ❤️ when I see how they celebrated classifying to the Olympics by beating the Canadian team 2-1. Regrettably they would end up losing the bronze medal match against the same team in Tokyo 2020.

View this post on Instagram

A post shared by Anissa Urtez (@nniiiss)

So I’m now filled with sadness to see how the people in power are asking to get them suspended for life and/or take legal action against them because they threw away some uniforms with the Mexican flag (ESPN). They recognized that it was a mistake.

View this post on Instagram

A post shared by Anissa Urtez (@nniiiss)

But I strongly think that this wasn’t their mistake only. It was a communication error from the administration if they didn’t explain to them the expectations that:

they had to return all the equipment with them without any logistics or financial support to do so
the bed covers were not a present like they thought
the uniforms hold such a strong emotional value, even those that they didn’t actually use during the Olympics

The administration should have defended them and supported them instead of using them as a piñata and a distraction. Now part of the public is dedicated to attacking them just like Trump attacked/attacks Mexicans (whether they are American or not): that they aren’t Mexicans, that they should leave, that no one wants them, etc. Even some Olympic athletes, that have represented or represent Mexico, have an attitude of: if it was hard for me, it should be hard for you too. That’s the same attitude that I see in the academic realm that is used to make minorities suffer when they already have to fight with less resources for the opportunities others have. It’s an attitude of crushing each other instead of supporting each other so we can fix and learn from our mistakes and grow together.

So my open letter to Anissa Urtez and the rest of the Mexican Women’s softball team from the Tokyo 2020 Olympics is:

Thank you Anissa for representing us with pride and for all the countless hours of effort and sacrifice that you’ve dedicated to this sport and this team.

All Mexicans are humans and humans make mistakes. At a distance it’s clear to me that there was an error in communication and in setting expectations from the part of the administration. If the administration wasn’t going to support you to cover the additional baggage expenses, if they expected you to carry back all your uniforms and equipment, if they knew about the mattress cover, etc, they should have told you so. They should have tried to protect you and your teammates as much as possible and they completely failed you.

You concentrated on performing well on your sport and what you know how to do better than anyone else in Mexico and among Mexican descendants in the US and elsewhere.

I’m really sad that after all you did for representing Mexico now they are using you as a piñata. That others are attacking you on social media and staining your memories of Tokyo 2020.

As a Mexican living in the US that spends part of his free time helping other Mexicans learn what I know how to do, I find it really emotional and motivating to read what it means to you to help guide the next generations in your sport where you are among the best in the world. Please keep it up! Hopefully this whole situation won’t discourage you and you’ll keep raising high the Mexican flag.

With support (well, hugs like we say in Mexico) from a random corner in the web in Baltimore.

You are a Mexican badass champion!!! 🇲🇽 💪🏽

I hope that Anissa and the team, as well as the future generations of the team, feels the same way that she felt not so long ago.

View this post on Instagram

A post shared by Anissa Urtez (@nniiiss)

Acknowledgments

This blog post was made possible thanks to:

References

[1] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

[2] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

[3] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Cleaning up my R packages and config files

Tue, 03 Nov 2020 00:00:00 +0000

Today was an unusual day at work given the US Elections. This meant that I had fewer meetings than what I’ve had lately. Earlier in the day I noticed an email announcing that the Bioconductor 3.13 docker image had been released for the next 6 month development cycle, which was a reminder of the recent Bioconductor 3.12 release. This prompted me to start updating my R packages.

In the past, I’ve updated all my currently installed R packages using the framework I described in a 2017 blog post. I remember seeing a tweet by Hadley Wickham not so long ago ¹ that for him, a new R version was an opportunity to start with a clean slate. I like having everything I need ready to use, but well, my list of installed R packages was getting pretty long. Given that 4 year windows of time are in our minds, it felt like a good opportunity to clean my house. Or well, my R packages.

Thus, I started writing down which are the packages I want to have installed. At this point for me, that includes several R/Bioconductor packages I’ve made and their dependencies in case I need to work on them to resolve bugs or add new features. My R packages already use many of my favorite R packages, so I took advantage of this in order to avoid having to list every single R package I like using. In order to achieve this, I used the dependencies = TRUE argument that you can use with remotes and BiocManager.

## Install from scratch
if (!requireNamespace("remotes", quietly = TRUE))
    install.packages("remotes")
remotes::install_cran("BiocManager")
BiocManager::version()

## Rprofile packages
remotes::install_github(c(
    "jalvesaq/colorout"
))
remotes::install_cran(c(
    "devtools",
    "usethis"
))

## Main packages
BiocManager::install(c(
    "biocthis",
    "brainflowprobes",
    "derfinder",
    "derfinderPlot",
    "GenomicState",
    "megadepth",
    "recount",
    "recountWorkflow",
    "recount3",
    "regutools",
    "regionReport",
    "spatialLIBD"
), dependencies = TRUE, update = FALSE)

Once I had my main packages, I started adding some from LIBD, some from CRAN, and other ones from Bioconductor. You can see the full list at my team’s website under Config files: R setup; R packages.

I was curious about how these changes affected my list of installed R packages and used my older 2017 blog post code to check this. That resulted in this list which shows 423 installed R packages and 589 that I used to have installed. I suspect that several of them will come back. For example, I needed to install blogdown to work on this blog post. Some of the 423 packages are new, like rsthemes which we recently learned about at the LIBD rstats club.

Today @lcolladotor went over new @rstudio & @Bioconductor developments that was started by tweets by @apreshill & @grrrck + new #rstats 📦s #recount3 #megadepth involving @chrisnwilks @dyzhang32 et al

📹https://t.co/E0DZ9Ej5qV
📓 https://t.co/FnDvcUmyUO
👀https://t.co/mEbQirhkF6 pic.twitter.com/BWByp9W7Wh
— LIBD rstats club (@LIBDrstats) October 3, 2020

Config files

Since I was doing all this work on both my macOS and Windows laptops for my R setup, I also went ahead and cleaned up a bit my configuration files. I have several of them with settings that I recommend others to use. That’s why I wrote a little “chapter” about them on my team’s website. The list includes:

Software I use (including R and RStudio)
R packages
R configuration files such as ~/.Rprofile and ~/.Renviron
Git configuration files ~/.gitconfig and ~/.gitignore_global
JHPCE (linux) configuration files such as ~/.bashrc, ~/.inputrc, ~/.bash_profile and ~/.sge_request

Wrapping up

I’m hoping that all this information will be useful to both current and new team members, but it could be useful also to you. Though you might need to adapt some things. Earlier on in my career I learned from how others use configuration files to speed up their work or make their work experience more enjoyable. I’m still learning, but now I have a decent bag of tricks to share too.

Have fun!

Image source

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[2\] A. Oleś. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.28.0. 2023. DOI: 10.18129/B9.bioc.BiocStyle. URL: https://bioconductor.org/packages/BiocStyle.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

\[4\] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

I couldn’t find the tweet right now. ↩︎

The start of a new phase for me

Mon, 21 Sep 2020 00:00:00 +0000

Today I have accepted a new role at the Lieber Institute for Brain Development (LIBD) as an Investigator. Since LIBD is affiliated with Johns Hopkins University (JHU), LIBD will support me on becoming a JHU faculty member ^[The rules are Department and School specific at JHU, which is something that I have yet to explore in detail.]. This change means that I’ll now be leading a team at LIBD, which is why I’m actively working on a new website ^[It’s technically a book made using bookdown.] called R/Bioconductor-powered Team Data Science. On that website I’m implementing the ideas I recently proposed on another type of data science group which involve Data Science guidance sessions based on my three year experience as a teaching assistant for the MPH capstone at JHBSPH.

I also plan to make publicly available as much material as we can. This will include a series of bootcamps that I’m going to start running today for my team members and guests. It will also include our team journal/software club presentations as well as the LIBD rstats club presentations. The material that I’ll be using was made by many collaborators and friends at JHU, Harvard, WEHI, RStudio and the rstats community in general. I’m really thankful that so many people have shared their educational materials in ways that can be re-used and for making it easier for all of us to learn from them. Thank you!

Beyond the 20% education and data science guidance efforts, my team will be working with our LIBD peers, JHU collaborators, and external allies on interesting research projects such as the teamSpatial ^[Keri Martinowich, Kristen Maynard, Stephanie Hicks, and their team members.] that developed #spatialLIBD, #recount3, and many other upcoming projects in the realms of RNA-seq, R/Bioconductor, and genomics. We will be developing methods and software, analyzing data, and aim to build a stronger bridge between LIBD and JHU.

I’m really thankful to everyone who has been a part of my journey: mentors, colleagues, allies, sponsors, and many friends. I want to highlight my mentors Jeff Leek and Andrew Jaffe without whom I wouldn’t be where I am right now. They’ve always been my fiercest advocates and have looked out for me all these years. It’s now my turn to try have a positive influence in others, though they set a very high bar! I have a lot to learn and work to do, but I’m looking forward to it. At LIBD, JHU and elsewhere I have many people I can ask for support, which I’m grateful for.

Finally, after reading Jeff Leek’s How to be a modern scientist book in preparation for the first bootcamp today, I’ve updated my Twitter username to match my GitHub one. That is, I’ve gone from fellgernon to lcolladotor. See you around on Twitter and elsewhere!

P.S. If you find this type of team and the work that we do appealing, let me know and we’ll evaluate what options are available =)

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

R/Bioconductor-powered Team Data Science was made possible thanks to bookdown and a GitHub Actions workflow that depends on Bioconductor’s docker image, which in turn depends on the Rocker Project.

bookdown (Xie, 2016)

References

[1] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

[2] A. Oleś. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.28.0. 2023. DOI: 10.18129/B9.bioc.BiocStyle. URL: https://bioconductor.org/packages/BiocStyle.

[3] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

[4] Y. Xie. bookdown: Authoring Books and Technical Documents with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2016. ISBN: 978-1138700109. URL: https://bookdown.org/yihui/bookdown.

[5] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

You just committed a large file and can't push to GitHub

Wed, 18 Mar 2020 00:00:00 +0000

Oh ohh! 😱 What do you do now?

The data me and my colleagues work with is typically too big for our personal computers, so we use a high performance computing environment (cluster) and mostly interact with it through the command line terminal. As you might know, I’m a big fan of version control and I use git plus GitHub for sharing our code ^[Between our personal computers and the JHPCE cluster, but also with collaborators and the community at large.]. That’s why I’ve been advocating others to use it for a while and when they do, they run to me if they have some issues. A while back, my former student Amy Peterson wrote a blog post titled git to know git: an 8 minute introduction which is useful if you are getting started. Amy also links to the excellent Happy Git and GitHub for the useR book.

Recurrent problem: you just commited a large file and can’t push to GitHub

One situation that I’ve frequently helped others with is when they use git add * or git add . and version control every file in their project. They then do a commit such as git commit -m "added all files" and run git push to sync their files to GitHub. But oops, GitHub complains that you are trying to commit files larger than 50 Mb and even grinds to a halt if they are larger than 100 Mb. Which given that we work with large data, happens frequently (even a PDF file can be that big!).

Ok, so what can you do at this point? Remember, this is the scenario where you just made that commit. That is, it’s the last commit. At that point, it’s best to undo your last git commit which is well described in this website. However, when you undo a commit, you can either fully wipe out any changes (wipe them out fully from your disk, not only git’s version control!) or undo the version control step but also keep your files intact. The main solution then is to use:

git reset --soft HEAD~1

However, maybe you tried other commands and it’s a bit more complicated than that. Which is why I greatly advise that you create a local backup of your main_project directory before you dive into commands such as git reset, specially whenever you see the --hard option being suggested. That is, do something like this:

## Nagivate to the parent directory of your main_project
cd directory_containing_your_project

## Check the full size of your project directory
du -sh main_project

## Do you have enough disk space?
df -h .

## If you have enough disk space, then create a full backup
cp -r main_project main_project_backup/

Once you are able to roll back the offending commit, instead of running git add * or git add . and similar commands, repeat the following cycle:

Check which files are not being version controlled (untracked) with git status.
Check how big each of your untracked files is. You can do so with ls -lh and ls -lh some_pattern.
Add the files or file patterns you want to avoid version controlling (the large files) to your .gitignore file ^[Note that you can also create .gitignore files inside each directory if you want to have tighter control. You could also ignore a full directory and the use git add -f to forcibly version control files, for example, echo "my_subdir" >> .gitignore plus git add -f my_subdir/*.R to forcibly version control the R script files inside my_subdir but ignore everything else.].
Double check that your pattern worked by confirming that these files do not show up as untracked when you run git status.

Repeat this until the only remaining untracked files are those you actually want to version control and that are small enough ^[If you really want to version control large files, look into git lfs.].

And that’s it! Keep version controlling your code and reap the benefits later on when you need to.

via GIPHY

We all run into this situation at some point (or multiple times), so please keep using version control. The benefits will outweigh the negatives!

Use case story: the issue

Thanks to a colleague who gave me permission to share their use case, here we can dive down into a real life example. First, this was their description (edited for anonymity):

Ran these, as per GitHub’s instructions, and it went fine

git init
git add README.md
git commit -m "first commit"
git remote add origin git@github.com:LieberInstitute/some_repository.git
git push -u origin master

Even added a .gitignore with some instructions on what to ignore when committing

But it wasn’t enough and I hadn’t come to appreciate yet that there’s no need to commit .rda’s or other very large files, so my git push died.

Since these were already staged, I thought the next move was to make another commit with an edited .gitignore listing anything in my rdas/

Putting me two branches ahead of master (Leo: commits I think)

I got frustrated and thought then that ok, I want to go back two commits…

my Googling suggested me to go for git reset --hard HEAD~2…

That’s when I started panicking 😭

My colleague started panicking at this point because they couldn’t see the files anymore. That is, running ls -lh rdas/ didn’t list the files they had worked on really hard to create over the past months. But at this point, these large files were under version control by git ^[Stored and hidden in some way inside the .git directory.], just not available on GitHub due to the file size limitations.

So then my panic Googling took me to https://stackoverflow.com/questions/5788037/recover-from-git-reset-hard

where I thought ok I just can run git reset HEAD@{2} , which was

7cb9bac HEAD@{0}: reset: moving to HEAD@{2}
1e8499d HEAD@{1}: reset: moving to HEAD~2
f03b884 HEAD@{2}: commit: Committing EVERYTHING
7cb9bac HEAD@{3}: commit: Commiting EVERYTHING			<- this one
1e8499d HEAD@{4}: commit (initial): first commit

And there it looked like not everything was quite lost, as I could see

$ git status
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#   (use "git push" to publish your local commits)
#
# Changes not staged for commit:
#   (use "git add/rm <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#	deleted:    .RData
#	deleted:    .gitignore
#	deleted:    *.R ## Lots of files with this pattern
#	deleted:    *.sh ## same story about the file pattern
#	deleted:    pdfs/*.pdf
#	deleted:    rdas/*.rda
#	deleted:    tables/*.csv
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	logs/
#	pdfs/*other*.pdf
#	rdas/*other*.rda
#	*other*.R
no changes added to commit (use "git add" and/or "git commit -a")

So now my colleague realizes that somehow git is version controlling the files, but the deleted label is still VERY scary!!

And then I thought ok I just need to re-stage those deleted files…

So I ran git add -A , but now I see

$ git status
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#   (use "git push" to publish your local commits)
#
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	deleted:    .RData
#	deleted:    .gitignore
#	deleted:    *.R ## Lots of files with this pattern
#	deleted:    *.sh ## same story about the file pattern
#	deleted:    pdfs/*.pdf
#	deleted:    rdas/*.rda
#	deleted:    tables/*.csv
#	new file:   logs/*.Rout
#	new file:   logs/*.sh.*
#	new file:   pdfs/*other*.pdf
#	new file:   rdas/*other*.rda
#	new file:   *other*.R
#	renamed:    pdfs/*something*.pdf -> pdfs/*something_else*.pdf
#	renamed:    rdas/*something*.rda -> rdas/*something_else*.rda

and clearly I don’t know what I’m doing so I stopped

And thanked the lord you were online. 😭😭

I promise I did some reviewing of resources and testing with local and JHPCE test dirs before

My colleague then pointed me to the directory with the files and we fixed their files.

Use case story: the solution

Like I mentioned earlier, the first thing to do in cases like this is to create a backup.

## Check how big it is
du -sh project_FINAL

## Create a backup
cp -r project_FINAL project_leo_backup/

## To wipe out the original copy
## proceed with EXTREME caution!
# rm -fr project_FINAL

## Then restore everything from your backup copy
cp -r project_leo_backup project_FINAL/

I actually messed up at one point and had to rely on this backup!! So, like I said,

Please backup everything before you start using git reset and similar commands!

Next, to undo all the git rm (deleting a file), I undid the git add -A step using a combination of git reset and git checkout (to restore files).

# https://stackoverflow.com/a/2125713/9374370
$ git reset HEAD
$ git checkout .

## check things
$ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)
nothing to commit, working tree clean
$ git log
commit 7cb9bac5378500b35a0c22480a5961248ecf67ea (HEAD -> master)
Author: xx <xx@jhmi.edu>
Date:   Tue Mar 17 19:27:31 2020 -0400
    Commiting EVERYTHING
commit 1e8499d7d41cab6c12ea23ccdb2da8120b00a7f7 (origin/master)
Author: XX <xx@jhmi.edu>
Date:   Tue Mar 17 19:04:03 2020 -0400
    first commit

$ ls -lh rdas
total 6.2G
## I see tons of stuff (many of which I'm now the "owner" of)

I then finally used git reset --soft to undo the last commit.

# https://www.git-tower.com/learn/git/faq/undo-last-commit
git reset --soft HEAD~1 ## note that I'm not using --hard

## Everything is back to before that big commit
$ git log
commit 1e8499d7d41cab6c12ea23ccdb2da8120b00a7f7 (HEAD -> master, origin/master)
Author: XX <xx@jhmi.edu>
Date:   Tue Mar 17 19:04:03 2020 -0400
    first commit

## and the files are there =)
$ ls -lh rdas
total 6.2G
-rwxrwx--- 1 lcollado lieber_jaffe 690M Mar 18 00:22 *.rda ## exmaple file

Now that the directory and files have been restored to before all files were committed, we can proceed to ignore large files. For example, we can ignore the rdas/ directory that has many large files that we don’t want to version control ^[Maybe later we’ll version control a few of them using git add -f rdas/some_file.rda but it’ll be a targeted version control command.].

$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   README.md

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	.RData
	.gitignore
	*.R ## Lots of files with this pattern
	*.sh ## Lots of files with this pattern
	pdfs/
	rdas/
	tables/

$ echo "rdas" >> .gitignore

## Notice that rdas is not there anymore ^^
$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   README.md

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	.RData
	.gitignore
	*.R ## Lots of files with this pattern
	*.sh ## Lots of files with this pattern
	pdfs/
	tables/

And now we can update our .gitignore and push this small change (ignoring the rdas/ directory) to GitHub.

$ git add .gitignore

$ git commit -m "Ignore rdas"
[master 52f1850] Ignore rdas
 1 file changed, 14 insertions(+)
 create mode 100755 .gitignore

$ git push
X11 forwarding request failed on channel 0
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 20 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 420 bytes | 210.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To github.com:LieberInstitute/some_repository.git
   1e8499d..52f1850  master -> master

And we are done!

via GIPHY

And I’ll get some free beers hehe 🍻😄

Omg you’re amazing beautiful lord n savior jesus christ

Your next 10 beers are on me.

plus 2 for the promotion 😄😇😎🙏😭

To which I replied

hehe, I’ve simply only have had more practice at this than you (fixing xx’s repos mostly hehe)

but yeah, backing up is the best thing you can do

that actually saved me from my one git reset --hard HEAD~1 command that should have been git reset --soft HEAD~1

Misc notes

Note that you might also want to use git status-size in some situations.

## From https://github.com/jtloong/git-status-size
$ git status-size

Finally, if you are a JHPCE user, I recommend including these lines in your ~/.bashrc file.

## Load the git module by default when qrsh/qsub
## thanks to Jiong Yang
if [[ $HOSTNAME == compute-* ]]; then
    echo "Adding LIBD modules"
    module use /jhpce/shared/jhpce/modulefiles/libd
    echo "Loading git"
    module load git
    module load git-status-size/github
    module load git-lfs/2.8.0
    module load rmate/1.5.9 ## macOS users
    module load conda_R/3.6.x ## default R version
fi

Acknowledgments

I greatly appreciate the anonymous user who reached out to me about this issue and had an excellent history of commands which allowed me to figure out a possible solution and then write this blog post (with their permission). We both hope that this information will be useful to ourselves and others in the future.

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

[1] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

[3] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Research Scientist: an academic career launch pad

Mon, 16 Mar 2020 00:00:00 +0000

After a long start to 2020 including the past four very busy weeks, I’m happy to announce that today March 16th 2020 I accepted a position as Research Scientist at the Lieber Institute for Brain Development in Baltimore, MD, USA.

via GIPHY

What will I do as a Research Scientist at LIBD?

At LIBD we currently have the following scientific ranks:

Research Technician
Research Assistant
Research Associate
Staff Scientist I, II and III
Research Scientist (+ Lead and Senior)
Investigator (+ Lead and Senior)

Research Scientists carry out research, do so scholarly, are tasked with being creative, are encouraged to seek funding, and can have supervisory and mentor roles.

In my agreement with LIBD, I will remain affiliated with the Data Science Team I led by Andrew E Jaffe while also having the opportunity to interact with and collaborate with fantastic biologists, data scientists, and researchers at LIBD, Johns Hopkins University, and beyond. Furthermore, I will officially help mentor LIBD scientists in data science and R tools. LIBD will support me while I build a small team around my research interests and prepare my first set of grant submissions. There will be a transition period and many things I need to learn: from requesting a budget to writing a job ad, as well as many details about grant requests. But ultimately, LIBD offered me a launch pad for an academic career.

via GIPHY

Details

One of my first tasks will be to refine the details of my role. For example, right now on the teaching side of things I’m imagining doing the following:

Lead a weekly LIBD rstats club meeting where I can go over topics common to several users.
Hold one-on-one data science guidance sessions, similar to the MPH capstone sessions I did during my teaching assistant years at JHBSPH.
Occasionally lead internal workshops, which could benefit from workshops I prepare for CDSB and elsewhere.
Get certified by RStudio and attend more R conferences & workshops.

From the scientific side, I will also learn more about the biological questions my LIBD colleagues are working on. These hypothesis-driven projects will provide me a new set of challenges and I will perform what me and my LCG-UNAM classmates were trained to do: to function as a bridge between multiple fields. I will also keep learning from my JHU colleagues and working with them as I refine my ideas. Currently, this is how I have framed my research interests:

My research aims to better understand the roots and signatures of disease (particularly psychiatric disorders) by zooming in across dimensions of gene activity: from studying gene expression at all feature levels (genes to exons to exon-exon junctions and un-annotated regions of expression), to using different gene expression measurement technologies (bulk RNA-seq, single cell/nuclei RNA-seq to spatial transcriptomics) that provide finer biological resolution and localization of gene expression. I’m interested in both hypothesis-driven projects as well as building general resources such as recount2 that enable us to contextualize our findings across all of the public human gene expression landscape. I use the R programming language for nearly all my work and like to organize my code in R packages that I share mostly through the Bioconductor project.

{width=800px}

Marvin Meyer

As for building my team, I know that overall one of the challenges everyone faces is recruiting talent. I’ll give this complicated process a shot by tapping into my network which includes many immigrants. I will also expand my network online and through conferences. Furthermore, I am committed to my outreach projects such as CDSB but will synergize them even more: re-use training materials, potentially recruit team members, and help me expand my network. In the meantime, if you are interested in working with me or my colleagues, please let me know!

Final decision

The decision I had to make was challenging, but in the end, I decided to stay at LIBD because they offered me a path to become a principal investigator in academia which it’s something I want to explore and see where it leads me. I will also formally be someone’s boss and gain managerial experience, work much closer with biologists in hypothesis-driven projects, and overall gain more experience and skills. My brother’s job tasks changed last week, to which I told him: “welcome to the real world”. Well, I guess that’s how it’ll be for me too.

Acknowledgments

I want to thank everyone that helped me through this decision-making process, provided me with information, asked me questions, answered questions, expanded my universe, advocated for me, advised me, cheered for me, and supported me. Thank you! I look forward to working with you!

via GIPHY

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

[1] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

[3] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Diving together into the unknown world of spatial transcriptomics

Sat, 29 Feb 2020 00:00:00 +0000

Yesterday was an extremely exciting day for me and my colleagues. We finished a project we had been working on and shared it with the world. Meaning, it’s done and we can relax for a little bit while we wait for feedback from our peers.

But this was not any project, at least not for me. Why do you ask? In general terms, it involved an analysis that you could not search on Google and find the answer for. That is, it involved diving into the unknown!

via GIPHY

The unknown is scary and as the lyrics say:

I’ve had my adventure, I don’t need something new I’m afraid of what I’m risking if I follow you Into the unknown

All of us have been building our careers with other types of data and/or experiments, and taking on a new type of data knowing we had an early access advantage over others was quite the challenge. I don’t know about my co-authors, but maybe some of them shared thoughts like mine that were along the lines: can I do this? can I make it work? do my analysis choices make sense? what will experts think of doing once they have access to this data? All while racing against time, even if it was just an illusion in our minds.

But it’s not my first adventure and I’ve picked up skills and confidence along the way. In particular, I’ve written Bioconductor R packages, dealt with pkgdown/travis issues like #1206, made shiny web applications, analyzed large RNA-seq data, written papers using GoogleDocs, gotten better at asking for help, among other skills.

It's Thurs but here's the "more" I promised https://t.co/4gahvWDbWE @travisci #pkgdown

+my fam of #rstats issues today https://t.co/iGFfHLtasM https://t.co/M8DZEHryJZ https://t.co/FU8qq3f5p6

I'm the pest that visits your GH issues here & there😅Nah, I put lots of 👨🏾‍💻into them!😊 https://t.co/1G0nd5ZwUJ
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) February 28, 2020

I’ve also gotten more comfortable with the idea that I can’t do it all. Others will shortly develop new methods for this type of data, or proper infrastructure to handle this data, or faster visualizations, and so goes on the list. But I’m proud and really happy to say that we built quite the robust prototype. Plus maybe we’ll be involved in shaping this future.

And you noticed that I mentioned we. That’s because I have been learning over the years how to foster collaborations. This particular project involved working with two other members of my workplace who are awesome and that I didn’t know that well. It also involved a new collaboration with someone I’ve known for a while now (we initially met through Twitter in 2014) but hadn’t had the chance to work with. Thus we dove into the unknown together 👩‍🚀🧑‍🚀.

via GIPHY

I feel like we complemented each other quite well and all I can confidently say that our new adventure so far has been very stimulating, even it cost me some sleep.

Woke up at 3:30 am dreaming about code 👨🏻‍💻 😴 and on an off till 5:30. Gave up and went to work early... hopefully I can remember the dreams 😅#CodeWhileDreaming #RestFuelsIdeas pic.twitter.com/h0mYHAaL5W
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) February 19, 2020

Spatial transcriptomics

So, where does spatial transcriptomics come into play and what does it mean? I work with gene activity data which we formally refer to as gene expression 🧬. That is, we measure 🔍🧮 the activity levels of genes for a particular biological condition or tissue sample. For several years now (about since 2007-2009) we have been able to measure many genes from a tissue sample, called bulk RNA-sequencing and abbreviated as RNA-seq.

That’s great! But biology is complicated and a single tissue sample is composed of multiple cells of various types. For example, in the brain there are cells that send signals around (neurons) and others that give structure to the brain. That is why technologies for measuring the gene expression at the single cell level were developed, abbreviated as scRNA-seq. scRNA-seq has been used widely to study mouse brains to live tissue samples.

In recent years I’ve been working with data from the human brain 🧠. The Lieber Institute for Brain Development has about two thousand brain samples. To preserve them for years to come, the brains are frozen 🥶. Cells are a bit fragile and freezing them breaks them. This fact has made it challenging to study data from frozen human brains. Several of my colleagues work on adapting research protocols to handle frozen human brain tissue. The research field overall has been able to generate single nucleus RNA sequencing (snRNA-seq) data and we are all generating some more.

snRNA-seq and scRNA-seq are great because you can measure what genes (pieces of the cell) are active, classify them into groups, and use prior knowledge to label these groups. However, you lose information about what part of the tissue they come from. That’s where technologies for spatial transcriptomics, that is, measuring gene expression 🧬 as close a possible to the single cell level yet retaining spatial coordinates are being actively developed. Thus, you end up with two main sources of data: the gene expression measurements but also images from the tissue (histology information). My coworkers anticipated what could these technologies be used for and what type of research questions they help us answer.

oh hey, that's us! stay tuned for a biorxiv preprint! @LieberInstitute @martinowk @kr_maynard @fellgernon @stephaniehicks @lmwebr https://t.co/uYpwQu52pS
— Andrew Jaffe (@andrewejaffe) February 24, 2020

Our project’s history

My coworkers got early access to a specific new type of spatial transcriptomics technology called Visium from the 10x Genomics company and started piloting it on human brain tissue. They recruited me to their project in early November 2019 (11th) and I recruited more colleagues in early December (4th). Today on February 28th 2020 we made public our research advances, code, and software we built for this project.

Given that we have many potential websites others can find us through, we decided to unify as much as possible the documentation even if that meant repeating it. The basic start of our documentation is included further below.

`spatialLIBD`

Welcome to the spatialLIBD project! It is composed of:

a shiny web application that we are hosting at spatial.libd.org/spatialLIBD/ that can handle a limited set of concurrent users,
a Bioconductor package at bioconductor.org/packages/spatialLIBD (or from here) that lets you analyze the data and run a local version of our web application (with our data or yours),
and a research article with the scientific knowledge we drew from this dataset. The analysis code for our project is available here and the high quality figures for the manuscript are available through Figshare.

The web application allows you to browse the LIBD human dorsolateral pre-frontal cortex (DLPFC) spatial transcriptomics data generated with the 10x Genomics Visium platform. Through the R/Bioconductor package you can also download the data as well as visualize your own datasets using this web application. Please check the manuscript or bioRxiv pre-print for more details about this project.

If you tweet about this website, the data or the R package please use the #spatialLIBD hashtag. You can find previous tweets that way as shown here. Thank you! Tweet #spatialLIBD

Study design

As a quick overview, the data presented here is from portion of the DLPFC that spans six neuronal layers plus white matter (A) for a total of three subjects with two pairs of spatially adjacent replicates (B). Each dissection of DLPFC was designed to span all six layers plus white matter (C). Using this web application you can explore the expression of known genes such as SNAP25 (D, a neuronal gene), MOBP (E, an oligodendrocyte gene), and known layer markers from mouse studies such as PCP4 (F, a known layer 5 marker gene).

Shiny website mirrors

Main shiny application website (note that the link must have a trailing slash / for it to work)
Shinyapps This version has less RAM memory but is typically deployed using the latest version of spatialLIBD.

R/Bioconductor package

The spatialLIBD package contains functions for:

Accessing the spatial transcriptomics data from the LIBD Human Pilot project (code on GitHub) generated with the Visium platform from 10x Genomics. The data is retrieved from Bioconductor’s ExperimentHub.
Visualizing the spot-level spatial gene expression data and clusters.
Inspecting the data interactively either on your computer or through spatial.libd.org/spatialLIBD/.

For more details, please check the documentation website or the Bioconductor package landing page here.

Installation instructions

Get the latest stable R release from CRAN. Then install spatialLIBD from Bioconductor using the following code:

if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")

BiocManager::install("spatialLIBD")

Access the data

Through the spatialLIBD package you can access the processed data in it’s final R format. However, we also provide a table of links so you can download the raw data we received from 10x Genomics.

Processed data

Using spatialLIBD you can access the Human DLPFC spatial transcriptomics data from the 10x Genomics Visium platform. For example, this is the code you can use to access the layer-level data. For more details, check the help file for fetch_data().

## Load the package
library('spatialLIBD')

## Download the spot-level data
spe <- fetch_data(type = 'spe')

## This is a SpatialExperiment object
spe

## class: SpatialExperiment 
## dim: 33538 47681 
## metadata(0):
## assays(2): counts logcounts
## rownames(33538): ENSG00000243485 ENSG00000237613 ... ENSG00000277475
##   ENSG00000268674
## rowData names(9): source type ... gene_search is_top_hvg
## colnames(47681): AAACAACGAATAGTTC-1 AAACAAGTATCTCCCA-1 ...
##   TTGTTTCCATACAACT-1 TTGTTTGTGTAAATTC-1
## colData names(69): sample_id Cluster ... array_row array_col
## reducedDimNames(6): PCA TSNE_perplexity50 ... TSNE_perplexity80
##   UMAP_neighbors15
## mainExpName: NULL
## altExpNames(0):
## spatialCoords names(2) : pxl_col_in_fullres pxl_row_in_fullres
## imgData names(4): sample_id image_id data scaleFactor

## Note the memory size
lobstr::obj_size(spe)

## 2.04 GB

## Remake the logo image with histology information
vis_clus(
    spe = spe,
    clustervar = 'layer_guess_reordered',
    sampleid = '151673',
    colors = libd_layer_colors,
    ... = ' DLPFC Human Brain Layers\nMade with github.com/LieberInstitute/spatialLIBD'
)

Citation

Below is the citation output from using citation('spatialLIBD') in R. Please run this yourself to check for any updates on how to cite spatialLIBD.

citation('spatialLIBD')

## To cite package 'spatialLIBD' in publications use:
## 
##   Pardo B, Spangler A, Weber LM, Hicks SC, Jaffe AE, Martinowich K,
##   Maynard KR, Collado-Torres L (2022). "spatialLIBD: an R/Bioconductor
##   package to visualize spatially-resolved transcriptomics data." _BMC
##   Genomics_. doi:10.1186/s12864-022-08601-w
##   <https://doi.org/10.1186/s12864-022-08601-w>,
##   <https://doi.org/10.1186/s12864-022-08601-w>.
## 
##   Maynard KR, Collado-Torres L, Weber LM, Uytingco C, Barry BK,
##   Williams SR, II JLC, Tran MN, Besich Z, Tippani M, Chew J, Yin Y,
##   Kleinman JE, Hyde TM, Rao N, Hicks SC, Martinowich K, Jaffe AE
##   (2021). "Transcriptome-scale spatial gene expression in the human
##   dorsolateral prefrontal cortex." _Nature Neuroscience_.
##   doi:10.1038/s41593-020-00787-0
##   <https://doi.org/10.1038/s41593-020-00787-0>,
##   <https://www.nature.com/articles/s41593-020-00787-0>.
## 
##   Huuki-Myers LA, Spangler A, Eagles NJ, Montgomergy KD, Kwon SH, Guo
##   B, Grant-Peters M, Divecha HR, Tippani M, Sriworarat C, Nguyen AB,
##   Ravichandran P, Tran MN, Seyedian A, Consortium P, Hyde TM, Kleinman
##   JE, Battle A, Page SC, Ryten M, Hicks SC, Martinowich K,
##   Collado-Torres L, Maynard KR (2023). "Integrated single cell and
##   unsupervised spatial transcriptomic analysis defines molecular
##   anatomy of the human dorsolateral prefrontal cortex." _bioRxiv_.
##   doi:10.1101/2023.02.15.528722
##   <https://doi.org/10.1101/2023.02.15.528722>,
##   <https://www.biorxiv.org/content/10.1101/2023.02.15.528722v1>.
## 
##   Kwon SH, Parthiban S, Tippani M, Divecha HR, Eagles NJ, Lobana JS,
##   Williams SR, Mark M, Bharadwaj RA, Kleinman JE, Hyde TM, Page SC,
##   Hicks SC, Martinowich K, Maynard KR, Collado-Torres L (2023).
##   "Influence of Alzheimer’s disease related neuropathology on local
##   microenvironment gene expression in the human inferior temporal
##   cortex." _bioRxiv_. doi:10.1101/2023.04.20.537710
##   <https://doi.org/10.1101/2023.04.20.537710>,
##   <https://www.biorxiv.org/content/10.1101/2023.04.20.537710v1>.
## 
## To see these entries in BibTeX format, use 'print(<citation>,
## bibtex=TRUE)', 'toBibtex(.)', or set
## 'options(citation.bibtex.max=999)'.

Please note that the spatialLIBD was only made possible thanks to many other R and bioinformatics software authors. We have cited their work either in the pre-print or the vignette of the R package.

Closing remarks

Overall, this project has everything that I like: R code, a Bioconductor package, challenging and interest biological data, excellent collaborator team, open communication, and so on.

Now, these are early days of the 10x Genomics Visium platform and there’s much we and others want to learn. So if you have the chance to hear anyone in our team talk more in detail about the project or you simply want to chat with them, here are some opportunities for you to do so as we’d love to collaborate with you or even hire you. Check Stephanie’s tweet and the LIBD career website for more details or simply get in touch with us.

Kristen R Maynard and me will present a The Scientist webinar on March 19th
Keri Martinowich will be at CVCSN 2020 March 26-27th
I’ll present a seminar at LIIGH-UNAM on March 30th
Kristen R Maynard will be at the 2020 Single Cell Symposium on April 20th
Likely Andrew E Jaffe and others will be at The Biology of Genomes 2020 May 5-9th
Stephanie Hicks will present at eRum 2020 May 27-30
Likely some of us will attend BioC2020 July 29-31

Finally, here’s the pre-print twitter thread:

🔥off the press! 👀 our @biorxivpreprint on human 🧠brain @LieberInstitute spatial 🌌🔬transcriptomics data 🧬using Visium @10xGenomics🎉#spatialLIBD

🔍https://t.co/RTW0VscUKR
👩🏾‍💻https://t.co/bsg04XKONr
📚https://t.co/FJDOOzrAJ6
📦https://t.co/Au5jwADGhY https://t.co/PiWEDN9q2N pic.twitter.com/aWy0yLlR50
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) February 29, 2020

Thank you for getting this far!

via GIPHY

References

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package                * version    date (UTC) lib source
##  AnnotationDbi            1.62.1     2023-05-08 [1] Bioconductor
##  AnnotationHub            3.8.0      2023-04-25 [1] Bioconductor
##  attempt                  0.3.1      2020-05-03 [1] CRAN (R 4.3.0)
##  backports                1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  beachmat                 2.16.0     2023-04-25 [1] Bioconductor
##  beeswarm                 0.4.0      2021-06-01 [1] CRAN (R 4.3.0)
##  benchmarkme              1.0.8      2022-06-12 [1] CRAN (R 4.3.0)
##  benchmarkmeData          1.0.4      2020-04-23 [1] CRAN (R 4.3.0)
##  bibtex                   0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  Biobase                * 2.60.0     2023-04-25 [1] Bioconductor
##  BiocFileCache            2.8.0      2023-04-25 [1] Bioconductor
##  BiocGenerics           * 0.46.0     2023-04-25 [1] Bioconductor
##  BiocIO                   1.10.0     2023-04-25 [1] Bioconductor
##  BiocManager              1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocNeighbors            1.18.0     2023-04-25 [1] Bioconductor
##  BiocParallel             1.34.2     2023-05-28 [1] Bioconductor
##  BiocSingular             1.16.0     2023-04-25 [1] Bioconductor
##  BiocStyle              * 2.28.0     2023-04-25 [1] Bioconductor
##  BiocVersion              3.17.1     2022-12-20 [1] Bioconductor
##  Biostrings               2.68.1     2023-05-16 [1] Bioconductor
##  bit                      4.0.5      2022-11-15 [1] CRAN (R 4.3.0)
##  bit64                    4.0.5      2020-08-30 [1] CRAN (R 4.3.0)
##  bitops                   1.0-7      2021-04-24 [1] CRAN (R 4.3.0)
##  blob                     1.2.4      2023-03-17 [1] CRAN (R 4.3.0)
##  blogdown                 1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown                 0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib                    0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem                   1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli                      3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  codetools                0.2-19     2023-02-01 [1] CRAN (R 4.3.1)
##  colorout                 1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  colorspace               2.1-0      2023-01-23 [1] CRAN (R 4.3.0)
##  config                   0.3.1      2020-12-17 [1] CRAN (R 4.3.0)
##  cowplot                  1.1.1      2020-12-30 [1] CRAN (R 4.3.0)
##  crayon                   1.5.2      2022-09-29 [1] CRAN (R 4.3.0)
##  curl                     5.0.1      2023-06-07 [1] CRAN (R 4.3.0)
##  data.table               1.14.8     2023-02-17 [1] CRAN (R 4.3.0)
##  DBI                      1.1.3      2022-06-18 [1] CRAN (R 4.3.0)
##  dbplyr                   2.3.2      2023-03-21 [1] CRAN (R 4.3.0)
##  DelayedArray             0.26.3     2023-05-28 [1] Bioconductor
##  DelayedMatrixStats       1.22.0     2023-05-08 [1] Bioconductor
##  digest                   0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  doParallel               1.0.17     2022-02-07 [1] CRAN (R 4.3.0)
##  dotCall64                1.0-2      2022-10-03 [1] CRAN (R 4.3.0)
##  dplyr                    1.1.2      2023-04-20 [1] CRAN (R 4.3.0)
##  dqrng                    0.3.0      2021-05-01 [1] CRAN (R 4.3.0)
##  DropletUtils             1.20.0     2023-05-08 [1] Bioconductor
##  DT                       0.28       2023-05-18 [1] CRAN (R 4.3.0)
##  edgeR                    3.42.4     2023-05-31 [1] Bioconductor
##  ellipsis                 0.3.2      2021-04-29 [1] CRAN (R 4.3.0)
##  evaluate                 0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  ExperimentHub            2.8.0      2023-04-25 [1] Bioconductor
##  fansi                    1.0.4      2023-01-22 [1] CRAN (R 4.3.0)
##  farver                   2.1.1      2022-07-06 [1] CRAN (R 4.3.0)
##  fastmap                  1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  fields                   14.1       2022-08-12 [1] CRAN (R 4.3.0)
##  filelock                 1.0.2      2018-10-05 [1] CRAN (R 4.3.0)
##  foreach                  1.5.2      2022-02-02 [1] CRAN (R 4.3.0)
##  generics                 0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  GenomeInfoDb           * 1.36.0     2023-05-08 [1] Bioconductor
##  GenomeInfoDbData         1.2.10     2023-05-06 [1] Bioconductor
##  GenomicAlignments        1.36.0     2023-04-25 [1] Bioconductor
##  GenomicRanges          * 1.52.0     2023-04-25 [1] Bioconductor
##  ggbeeswarm               0.7.2      2023-04-29 [1] CRAN (R 4.3.0)
##  ggplot2                  3.4.2      2023-04-03 [1] CRAN (R 4.3.0)
##  ggrepel                  0.9.3      2023-02-03 [1] CRAN (R 4.3.0)
##  glue                     1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  golem                    0.4.1      2023-06-05 [1] CRAN (R 4.3.0)
##  gridExtra                2.3        2017-09-09 [1] CRAN (R 4.3.0)
##  gtable                   0.3.3      2023-03-21 [1] CRAN (R 4.3.0)
##  HDF5Array                1.28.1     2023-05-01 [1] Bioconductor
##  highr                    0.10       2022-12-22 [1] CRAN (R 4.3.0)
##  htmltools                0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  htmlwidgets              1.6.2      2023-03-17 [1] CRAN (R 4.3.0)
##  httpuv                   1.6.11     2023-05-11 [1] CRAN (R 4.3.0)
##  httr                     1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  interactiveDisplayBase   1.38.0     2023-04-25 [1] Bioconductor
##  IRanges                * 2.34.0     2023-05-08 [1] Bioconductor
##  irlba                    2.3.5.1    2022-10-03 [1] CRAN (R 4.3.0)
##  iterators                1.0.14     2022-02-05 [1] CRAN (R 4.3.0)
##  jquerylib                0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite                 1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  KEGGREST                 1.40.0     2023-04-25 [1] Bioconductor
##  knitcitations          * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr                    1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  labeling                 0.4.2      2020-10-20 [1] CRAN (R 4.3.0)
##  later                    1.3.1      2023-05-02 [1] CRAN (R 4.3.0)
##  lattice                  0.21-8     2023-04-05 [1] CRAN (R 4.3.1)
##  lazyeval                 0.2.2      2019-03-15 [1] CRAN (R 4.3.0)
##  lifecycle                1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  limma                    3.56.2     2023-06-04 [1] Bioconductor
##  lobstr                   1.1.2      2022-06-22 [1] CRAN (R 4.3.0)
##  locfit                   1.5-9.8    2023-06-11 [1] CRAN (R 4.3.0)
##  lubridate                1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magick                   2.7.4      2023-03-09 [1] CRAN (R 4.3.0)
##  magrittr                 2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  maps                     3.4.1      2022-10-30 [1] CRAN (R 4.3.0)
##  Matrix                   1.5-4.1    2023-05-18 [1] CRAN (R 4.3.1)
##  MatrixGenerics         * 1.12.0     2023-05-08 [1] Bioconductor
##  matrixStats            * 1.0.0      2023-06-02 [1] CRAN (R 4.3.0)
##  memoise                  2.0.1      2021-11-26 [1] CRAN (R 4.3.0)
##  mime                     0.12       2021-09-28 [1] CRAN (R 4.3.0)
##  munsell                  0.5.0      2018-06-12 [1] CRAN (R 4.3.0)
##  paletteer                1.5.0      2022-10-19 [1] CRAN (R 4.3.0)
##  pillar                   1.9.0      2023-03-22 [1] CRAN (R 4.3.0)
##  pkgconfig                2.0.3      2019-09-22 [1] CRAN (R 4.3.0)
##  plotly                   4.10.2     2023-06-03 [1] CRAN (R 4.3.0)
##  plyr                     1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  png                      0.1-8      2022-11-29 [1] CRAN (R 4.3.0)
##  prettyunits              1.1.1      2020-01-24 [1] CRAN (R 4.3.0)
##  promises                 1.2.0.1    2021-02-11 [1] CRAN (R 4.3.0)
##  purrr                    1.0.1      2023-01-10 [1] CRAN (R 4.3.0)
##  R.methodsS3              1.8.2      2022-06-13 [1] CRAN (R 4.3.0)
##  R.oo                     1.25.0     2022-06-12 [1] CRAN (R 4.3.0)
##  R.utils                  2.12.2     2022-11-11 [1] CRAN (R 4.3.0)
##  R6                       2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  rappdirs                 0.3.3      2021-01-31 [1] CRAN (R 4.3.0)
##  RColorBrewer             1.1-3      2022-04-03 [1] CRAN (R 4.3.0)
##  Rcpp                     1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RCurl                    1.98-1.12  2023-03-27 [1] CRAN (R 4.3.0)
##  RefManageR               1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rematch2                 2.1.2      2020-05-01 [1] CRAN (R 4.3.0)
##  restfulr                 0.0.15     2022-06-16 [1] CRAN (R 4.3.0)
##  rhdf5                    2.44.0     2023-04-25 [1] Bioconductor
##  rhdf5filters             1.12.1     2023-04-30 [1] Bioconductor
##  Rhdf5lib                 1.22.0     2023-04-25 [1] Bioconductor
##  rjson                    0.2.21     2022-01-09 [1] CRAN (R 4.3.0)
##  rlang                    1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown                2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  Rsamtools                2.16.0     2023-04-25 [1] Bioconductor
##  RSQLite                  2.3.1      2023-04-03 [1] CRAN (R 4.3.0)
##  rstudioapi               0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  rsvd                     1.0.5      2021-04-16 [1] CRAN (R 4.3.0)
##  rtracklayer              1.60.0     2023-04-25 [1] Bioconductor
##  S4Arrays                 1.0.4      2023-05-14 [1] Bioconductor
##  S4Vectors              * 0.38.1     2023-05-02 [1] Bioconductor
##  sass                     0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  ScaledMatrix             1.8.1      2023-05-03 [1] Bioconductor
##  scales                   1.2.1      2022-08-20 [1] CRAN (R 4.3.0)
##  scater                   1.28.0     2023-04-25 [1] Bioconductor
##  scuttle                  1.10.1     2023-05-02 [1] Bioconductor
##  sessioninfo            * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  shiny                    1.7.4      2022-12-15 [1] CRAN (R 4.3.0)
##  shinyWidgets             0.7.6      2023-01-08 [1] CRAN (R 4.3.0)
##  SingleCellExperiment   * 1.22.0     2023-04-25 [1] Bioconductor
##  spam                     2.9-1      2022-08-07 [1] CRAN (R 4.3.0)
##  sparseMatrixStats        1.12.0     2023-05-08 [1] Bioconductor
##  SpatialExperiment      * 1.10.0     2023-04-25 [1] Bioconductor
##  spatialLIBD            * 1.13.4     2023-05-24 [1] Github (LieberInstitute/spatialLIBD@edc8b72)
##  statmod                  1.5.0      2023-01-06 [1] CRAN (R 4.3.0)
##  stringi                  1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr                  1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  SummarizedExperiment   * 1.30.2     2023-06-11 [1] Bioconductor
##  tibble                   3.2.1      2023-03-20 [1] CRAN (R 4.3.0)
##  tidyr                    1.3.0      2023-01-24 [1] CRAN (R 4.3.0)
##  tidyselect               1.2.0      2022-10-10 [1] CRAN (R 4.3.0)
##  timechange               0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  utf8                     1.2.3      2023-01-31 [1] CRAN (R 4.3.0)
##  vctrs                    0.6.3      2023-06-14 [1] CRAN (R 4.3.0)
##  vipor                    0.4.5      2017-03-22 [1] CRAN (R 4.3.0)
##  viridis                  0.6.3      2023-05-03 [1] CRAN (R 4.3.0)
##  viridisLite              0.4.2      2023-05-02 [1] CRAN (R 4.3.0)
##  withr                    2.5.0      2022-03-03 [1] CRAN (R 4.3.0)
##  xfun                     0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  XML                      3.99-0.14  2023-03-19 [1] CRAN (R 4.3.0)
##  xml2                     1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  xtable                   1.8-4      2019-04-21 [1] CRAN (R 4.3.0)
##  XVector                  0.40.0     2023-04-25 [1] Bioconductor
##  yaml                     2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
##  zlibbioc                 1.46.0     2023-04-25 [1] Bioconductor
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Learning from our search history

Wed, 12 Feb 2020 00:00:00 +0000

Origin of the idea

Recently the team I work with has had a few new members and I’ve been thinking lately of ways we could try to help them. The team leader was traveling this week, which gave me the opportunity to come up with a new type of session and test it out. That’s the origin of this learning from our search history idea. We tested it today and I’m quite happy with the results so far, so I thought it would be useful to document what we did and share it with others.

Motivation

The theory

As I show in the slides below, in our group we follow the you must try, and then you must ask framework although with a little different interpretation. The basic goal is to search independently for a period of time (say 15 minutes), but then ask others for help if you are still stuck beyond that point.

In other words, you have to be able to find some answers yourself since that’s part of our job using resources that range from Google Search, to the Bioconductor Support, to the RStudio Community, among other websites. However, we also acknowledge that some questions have difficult answers. Maybe a Stack Overflow thread has multiple answers and not necessarily the top voted question has the answer you are looking for. So instead of spending too much energy, we also tell our members to avoid getting into a rabbit hole for hours. That’s where asking for help comes to play. And you can ask for help from any community you belong to: those involved in the project through Slack, our full team, other scientists in our university, communities we belong to like the rstats community on Twitter, the Bioconductor users community at large, etc.

I did mention that it’s ideal to think about the person who will be helping you answer your question. Small reproducible examples, versions of the software you used, sharing your code on GitHub ¹, the commands you used and the order you used them in can all be valuable for resolving different types of problems. Jenny Bryan has talked much more about this subject, for example in this 2018 webinar.

In practice

Trying for a while then asking for help is all good in practice. However, asking for help is very challenging. It’s scary, you open yourself because you show what you don’t know to other people, and sadly historically many questions have been met with negative feedback on the Internet. Thus, they can make you feel dumb, stupid and many other negative emotions.

I do think that asking for help can be worth it and even wrote a previous blog post on this subject. Some reasons why it’s worth it include being able to move on with your work ², you might learn something new, and if you follow the strategies for helping others help you, you might even figure out the answer yourself.

Now, we all ask for help regardless of how long we have been writing code. Here’s an example tweet that conveys the same message. There are thousands of such tweets online.

I conduct approximately 3948234 programming-related Google searches per day. So does every other developer I know, whether they have 5 weeks of experience or 5 years. Help normalize this practice by tweeting your daily searches with #devgoogle! https://t.co/XfNVUZlhC5
— Lyzi Diamond (@lyzidiamond) January 15, 2019

We are diverse

The team (shown as of May 2018) we work at is very diverse because we all:

have different backgrounds,
have acquired different skills,
are at different career stages (from rotation student up to associate professor),
have different interests,
use different operating systems (from Fedora to Ubuntu to macOS to Windows)
use different tools (mobaxterm vs putty, TextMate vs RStudio, …).

But also because we seek help in different ways ³ and we learn differently.

This means that we have a lot to learn from each other 😊🤓.

Learn from each other exercise

At bit.ly/learnfromsearch you can find a copy of the Google Spreadsheet with the information you need for your team.

Some rules

First, we need to make sure that everyone will feel save to ask questions. That’s why I:

reminded others about our code of conduct,
invited everyone to practice their empathy and be mindful that language matters ⁴,
to keep everyone’s time in their mind as a question could lead to a longer discussion which is best to take another occasion. ⁵

Main steps

The idea is that you pick a problem you solved recently and share:

what you were trying to solve,
the actual text you searched for in Google or elsewhere ⁶,
the link of the website where you find your answer or that guided in this process.

We improved the steps as we were testing this! 🙌🏽🙂

Once everyone has contributed their information to the spreadsheet, we then proceed to go around the table showing and explaining our search use cases.

Goals

Ultimately the goals of this exercise are to

empower ourselves with the knowledge from our teammates,
learn from how we all find help,
build a supportive environment where we feel comfortable asking for help.

Thus in the end, we are enabling our team to fully follow the you must try, and then you must ask framework.

Test session

The first and only session so far went approximately like this:

Min 0-5: get settled in the room.
Min 5-22: presentation about the idea to get members to buy into it.
Min 22-26: demonstration.
Min 26-33: members prepared their information to share with the team.
Min 33-58: members presented their problems, the searches the did, and the solution(s) they found.
Min 58-60: quick wrap up.

My search example

At bit.ly/learnfromsearch I left some examples (anonymized). Mine was about increasing the point size of the output of a plot made with scater::plotReducedDim() which returns a ggplot2 (Wickham, 2016) object. Hence why I searched in Google for ggplot2 increase point size which lead me to the geom_point() reference website. I then tried using + geom_point(size = 20) but that broke the color mapping. I was about to dive into the GitHub code for scater (McCarthy, Campbell, Lun, and Willis, 2017) as this my go-to behavior for many similar quests, but I decided to check the help page using ?scater::plotReducedDim.

… Additional arguments for visualization, see ?“scater-plot-args” for details.

The documentation for ... lead me to ?"scater-plot-args" where I finally found the point_size argument

point_size: Numeric scalar, specifying the size of the points. Defaults to NULL.

and that solved my problem.

So what used to look like this:

library('scater')
library('SummarizedExperiment')

plotReducedDim(
    sce,
    dimred = 'PCA',
    colour_by = 'my_cluster_variable',
    theme_size = 20
)

now looks like this:

plotReducedDim(
    sce,
    dimred = 'PCA',
    colour_by = 'my_cluster_variable',
    theme_size = 20,
    point_size = 5
)

Other examples involve different websites and showcase the diversity of questions we have as a team.

Conclusions

I hope that you like this idea and try it out yourselves. Maybe some of the lessons you learn trying it out could be useful to us as well. Ultimately, the information stored there could be useful for new team members as well as for current members since the spreadsheet becomes like an informal FAQ or team wiki.

via GIPHY

I was strongly encouraged by the feedback two members gave me individually after our trial session. Maybe this is not for everyone as we realized that it’s quite hard to be anonymous while participating ⁷. This idea could evolve into something else, but at least for today, I’m happy with the amount of people that bought into the trial and participated in it. We’ll see what happens next.

Acknowledgments

This blog post was made possible thanks to:

I would also like to acknowledge the general inspiration I’ve gotten from Alison Hill’s educational work. Once the rstudio::conf(2020) videos are available, check the work her intern Desirée de Leon showcased which is related to the following tweet.

Ever seen a giraffe that can fit in a teacup? Time to share the first draft of the R and stats project that Hasse Wallum and I have been working on for some time. More to come! #rstats 🦒 https://t.co/SOlBzop8vz
— Desirée De Leon (@dcossyle) August 14, 2019

P.S. dinámica in Spanish is used for a set of exercises that have a specific purpose in mind. I now realize that dynamic doesn’t hold the same meaning. Oh well 🤷🏾

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[2\] D. J. McCarthy, K. R. Campbell, A. T. L. Lun, and Q. F. Willis. “Scater: pre-processing, quality control, normalisation and visualisation of single-cell RNA-seq data in R”. In: Bioinformatics 33 (8 2017), pp. 1179-1186. DOI: 10.1093/bioinformatics/btw777.

\[3\] M. Morgan, V. Obenchain, J. Hester, and H. Pagès. SummarizedExperiment: SummarizedExperiment container. R package version 1.30.2. 2023. DOI: 10.18129/B9.bioc.SummarizedExperiment. URL: https://bioconductor.org/packages/SummarizedExperiment.

\[4\] A. Oleś. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.28.0. 2023. DOI: 10.18129/B9.bioc.BiocStyle. URL: https://bioconductor.org/packages/BiocStyle.

\[5\] H. Wickham. ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, 2016. ISBN: 978-3-319-24277-4. URL: https://ggplot2.tidyverse.org.

\[6\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

\[7\] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package              * version    date (UTC) lib source
##  backports              1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  beachmat               2.16.0     2023-04-25 [1] Bioconductor
##  beeswarm               0.4.0      2021-06-01 [1] CRAN (R 4.3.0)
##  bibtex                 0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  Biobase              * 2.60.0     2023-04-25 [1] Bioconductor
##  BiocGenerics         * 0.46.0     2023-04-25 [1] Bioconductor
##  BiocManager            1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocNeighbors          1.18.0     2023-04-25 [1] Bioconductor
##  BiocParallel           1.34.2     2023-05-28 [1] Bioconductor
##  BiocSingular           1.16.0     2023-04-25 [1] Bioconductor
##  BiocStyle            * 2.28.0     2023-04-25 [1] Bioconductor
##  bitops                 1.0-7      2021-04-24 [1] CRAN (R 4.3.0)
##  blogdown               1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown               0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib                  0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem                 1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli                    3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  codetools              0.2-19     2023-02-01 [1] CRAN (R 4.3.1)
##  colorout               1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  colorspace             2.1-0      2023-01-23 [1] CRAN (R 4.3.0)
##  crayon                 1.5.2      2022-09-29 [1] CRAN (R 4.3.0)
##  DelayedArray           0.26.3     2023-05-28 [1] Bioconductor
##  DelayedMatrixStats     1.22.0     2023-05-08 [1] Bioconductor
##  digest                 0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  dplyr                  1.1.2      2023-04-20 [1] CRAN (R 4.3.0)
##  evaluate               0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fansi                  1.0.4      2023-01-22 [1] CRAN (R 4.3.0)
##  fastmap                1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics               0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  GenomeInfoDb         * 1.36.0     2023-05-08 [1] Bioconductor
##  GenomeInfoDbData       1.2.10     2023-05-06 [1] Bioconductor
##  GenomicRanges        * 1.52.0     2023-04-25 [1] Bioconductor
##  ggbeeswarm             0.7.2      2023-04-29 [1] CRAN (R 4.3.0)
##  ggplot2              * 3.4.2      2023-04-03 [1] CRAN (R 4.3.0)
##  ggrepel                0.9.3      2023-02-03 [1] CRAN (R 4.3.0)
##  glue                   1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  gridExtra              2.3        2017-09-09 [1] CRAN (R 4.3.0)
##  gtable                 0.3.3      2023-03-21 [1] CRAN (R 4.3.0)
##  htmltools              0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr                   1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  IRanges              * 2.34.0     2023-05-08 [1] Bioconductor
##  irlba                  2.3.5.1    2022-10-03 [1] CRAN (R 4.3.0)
##  jquerylib              0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite               1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations        * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr                  1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lattice                0.21-8     2023-04-05 [1] CRAN (R 4.3.1)
##  lifecycle              1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate              1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr               2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  Matrix                 1.5-4.1    2023-05-18 [1] CRAN (R 4.3.1)
##  MatrixGenerics       * 1.12.0     2023-05-08 [1] Bioconductor
##  matrixStats          * 1.0.0      2023-06-02 [1] CRAN (R 4.3.0)
##  munsell                0.5.0      2018-06-12 [1] CRAN (R 4.3.0)
##  pillar                 1.9.0      2023-03-22 [1] CRAN (R 4.3.0)
##  pkgconfig              2.0.3      2019-09-22 [1] CRAN (R 4.3.0)
##  plyr                   1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6                     2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp                   1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RCurl                  1.98-1.12  2023-03-27 [1] CRAN (R 4.3.0)
##  RefManageR             1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang                  1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown              2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi             0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  rsvd                   1.0.5      2021-04-16 [1] CRAN (R 4.3.0)
##  S4Arrays               1.0.4      2023-05-14 [1] Bioconductor
##  S4Vectors            * 0.38.1     2023-05-02 [1] Bioconductor
##  sass                   0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  ScaledMatrix           1.8.1      2023-05-03 [1] Bioconductor
##  scales                 1.2.1      2022-08-20 [1] CRAN (R 4.3.0)
##  scater               * 1.28.0     2023-04-25 [1] Bioconductor
##  scuttle              * 1.10.1     2023-05-02 [1] Bioconductor
##  sessioninfo          * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  SingleCellExperiment * 1.22.0     2023-04-25 [1] Bioconductor
##  sparseMatrixStats      1.12.0     2023-05-08 [1] Bioconductor
##  stringi                1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr                1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  SummarizedExperiment * 1.30.2     2023-06-11 [1] Bioconductor
##  tibble                 3.2.1      2023-03-20 [1] CRAN (R 4.3.0)
##  tidyselect             1.2.0      2022-10-10 [1] CRAN (R 4.3.0)
##  timechange             0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  utf8                   1.2.3      2023-01-31 [1] CRAN (R 4.3.0)
##  vctrs                  0.6.3      2023-06-14 [1] CRAN (R 4.3.0)
##  vipor                  0.4.5      2017-03-22 [1] CRAN (R 4.3.0)
##  viridis                0.6.3      2023-05-03 [1] CRAN (R 4.3.0)
##  viridisLite            0.4.2      2023-05-02 [1] CRAN (R 4.3.0)
##  withr                  2.5.0      2022-03-03 [1] CRAN (R 4.3.0)
##  xfun                   0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2                   1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  XVector                0.40.0     2023-04-25 [1] Bioconductor
##  yaml                   2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
##  zlibbioc               1.46.0     2023-04-25 [1] Bioconductor
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

So you can link to specific lines and see things you changed through time that might be the source of the problem are among the main reasons why you should try to version control your code. ↩︎
Potentially to a more interesting problem than the one you are stuck currently at. ↩︎
For example, some use a particular project Slack channel, others the lab one, others through direct messages. ↩︎
If you say that something is obvious or easy, you are telling the other person that it should be easy for them too, but we know that it isn’t the case and that’s why they are asking a question. ↩︎
If we want to incorporate this exercise into our weekly meetings (maybe once a month), we need to make sure that our team meeting will finish on time. ↩︎
The use of keywords can dramatically affect a search results, and this information is useful to convey among ourselves so we can learn to search for help more effectively. ↩︎
Basically, you can only be anonymous for those not in the room at the time the question was discussed. ↩︎

Conference feelings: from newbie to sponsor

Mon, 03 Feb 2020 00:00:00 +0000

In the summer of 2008, nearly 12 years ago, I attended my first R/Bioconductor conference: BioC2008. Just last week I went to my second rstudio::conf(2020) which I greatly enjoyed. After some tweets exchanges today, I started reflecting on my journey and wanted to share my thoughts.

Why I like going to conferences

I typically enjoy going to conferences, though I also end up exhausted.

via GIPHY

Part of it could be the traveling and all that goes with it, but I think that conferences are mostly mentally taxing. There’s an information deluge which I typically find overwhelming. Sure, I love finding about new work but at some point it’s like collecting bookmarks of stuff to read or try out later ¹. Yet the most exhausting component of going to a conference could be the networking aspect. Meeting others and catching up with friends is extremely rewarding which is why I ultimately sign up for the next conference. How rewarding can it be? My most extreme example is when two professors invited me to apply to the Ph.D. program that I ended up studying. Though simply making friends and having friendly faces around you can go a long way. I’ll expand a bit on this later.

As a newbie

Across my career I have felt as a newbie when attending many different conferences. I might be alone or maybe have a friend at the conference, or it could be my first time at a conference that I haven’t been to. Your mind, like mine, can be filled up with questions that later might seem simple like:

How should I dress?
Is it ok to drink alcohol? ²
How do I not feel alone?
Am I the odd one here?
How can I talk to X or Y person I’ve heard about before but never met?

My answers have likely changed over time and currently are:

Comfortable, but remember that this is a work event. So look professional by getting a haircut and dressing up a bit ³.
Yes, but not too much and if I don’t want any, that’s ok too.
Talk to whoever seems friendly: could be someone new or an old friend or acquaintance.
Maybe, but meh 🤷🏽‍♂️. I do try talking to other latinos or foreigners specially when there’s not many of us around.
This one is still hard, but maybe someone I know knows them and can introduce me. If so, I ask for the favor. Otherwise I try to approach them with a question about their work.

The desire to avoid feeling alone is something that can still shape my behavior at a conference. If possible, I like having some close friends I can hang out with and be relaxed, almost as if we were not at the conference. They provide a zone that I can enter, recharge, and then head out of it again to continue networking.

Yet recently, I’ve grown more conscious of the fact that I’ve been around longer than others. Meaning that I might be the person that knows X or Y that they want to meet or talk to. That is, I find myself more frequently in the position of introducing people and helping them make connections. Somehow asking X or Y: “hey, do you have a minute? Can I introduce you to someone?” is way less scary even if I’m not the closest acquaintance of X or Y. That is, becoming a sponsor: mostly the access portion as defined by this blog post by Emily Robinson.

So, while I still network like I used to, a portion of my time is spent checking on others I know so they are not alone and helping them when possible make the connections they want. Doing so also helps me strengthen my own connections as I detailed near the end of my tweet series post-rstudio::conf(2020).

#rstudioconf + I also asked & appreciated their advise! Thanks!

I'm just trying to emulate what others have done for me in the past (too many to list, thank youuu!), as I know that getting someone to answer the doorbell is the first step towards opening a door🤗🙌🏾#rstats 11/11
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) February 2, 2020

Feeling empowered

Reaching out and trying to meet someone is hard and scary. Something that helped me since BioC2008 was the fact that I had others in mind: I knew that I was going to teach R later that fall at LCG-UNAM, so any questions I could get through or people I could meet were huge wins for me. Thus, thinking about others empowered me to network. And that’s still true for me through our CDSB project at comunidadbioinfo.github.io.

If I’m still feeling shy or alone, I remember a story from my dad: he claims that without approaching others who were having dinner at a table, introducing himself and creating a social connection, his ideas would have not made the Escherichia coli genome paper that helped launch his career. Personally, once I break that barrier and start to feel comfortable, I feel like I can ask anything: specially feedback and support from others on how to advance some ideas of mine. Lately, that’s mostly been related to our CDSB project.

I wish others could feel empowered too, yet it’s hard to get there. For now, I simply try to help others emulating what others have done for me. Ultimately, I know that getting someone to answer the doorbell is the first step towards opening a door 🤗🙌🏾 which motivates me and helps me overcome my shyness. Whatever they do after someone answers the doorbell is up to the person I’m helping. Yet in that moment, the satisfaction of helping others make connections fulfills me. Later on, it brings a smile to my face even if I’m exhausted after the conference.

via GIPHY

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Which is why I have recently taken to Twitter to take “notes”. Notes that I make public and not necessarily go back and revisit, but if I want to, I know where they are. ↩︎
Or not drink alcohol. Whichever you want. ↩︎
I don’t wear suits or anything like that though. ↩︎

Windows user space issues with installing R packages

Wed, 18 Sep 2019 00:00:00 +0000

Are you a Microsoft Windows R user? Does your Windows username include a space? Like Firstname Lastname. Then you might occassionally run into issues installing packages due to spaces.

Solutions

You could either re-install Windows with a username that has no spaces such as Lastname ^[This is the case in the Bioconductor Windows build machine where the username is biocbuild as you can see here.], but that’s probably not an easy option. Or you can:

Edit your TMP and TEMP environment variables to a location with no spaces, like C:\TEMP following instructions like these ones.
Preferably install R at a location with no spaces, like C:\R, instead of the default C:\Program Files ^[In the Bioconductor Windows build machines there are again no spaces in the path to the R installation and the library where packages are installed.].

Backstory

A co-worker wanted to install the clusterprofiler Bioconductor package which depends on the DO.db Bioconductor package. This co-worker uses a Windows machine that has a username with a space. Let’s say it was me with Leo Collado to keep them anonymous. The DO.db is only available as a “Source” package with no Windows binary as you can see here.

![](http://lcolladotor.github.io/post/2019-09-18-windows-user-space-issues-with-installing-r-packages_files/Screen Shot 2019-09-17 at 10.33.47 PM.png){width=600px}

This means that R has to:

download a tar.gz file,
uncompress it,
and then install it.

In particular, we are talking about DO.db_2.9.tar.gz in this case.

The installation instructions for DO.db are:

if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")

BiocManager::install("DO.db")

Uncompressing

Internally, BiocManager::install() ends up using utils::install.packages(). The first step, downloading, works well. Uncompressing a file in this scenario fails. Why?

> BiocManager::install('DO.db', lib = 'C:/R/R-3.6.0/library')
Bioconductor version 3.9 (BiocManager 1.30.4), R 3.6.0 (2019-04-26)
Installing package(s) 'BiocVersion', 'DO.db'

## removed output

trying URL 'https://bioconductor.org/packages/3.9/data/annotation/src/contrib/DO.db_2.9.tar.gz'
Content type 'application/x-gzip' length 1769978 bytes (1.7 MB)
downloaded 1.7 MB

Error in untar2(tarfile, files, list, exdir, restore_times) :
  incomplete block on file

The downloaded source packages are in
        ‘C:\Users\Leo Collado\AppData\Local\Temp\RtmpqiBJ53\downloaded_packages’

If you search on Google the error message you’ll find links like this one which hint towards an incomplete download. But the download works. You can even download the file and try to run untar() manually and it will fail.

We were told to try installing R at a location with no spaces, so by this point, R was installed at C:\R\R-3.6.0\, hence the lib specification you see above, though it’s irrelevant for these errors.

Uncompressing the tar.gz file is done by utils::untar(). If you look at the code for utils::untar() you’ll see:

## The function definition of utils::untar
function (tarfile, files = NULL, list = FALSE, exdir = ".", compressed = NA, 
  extras = NULL, verbose = FALSE, restore_times = TRUE,
  support_old_tars = Sys.getenv("R_SUPPORT_OLD_TARS", 
    FALSE), tar = Sys.getenv("TAR")) 

## Inside utils::untar()
if (inherits(tarfile, "connection") || identical(tar, "internal")) {
  if (!missing(compressed)) 
    warning("argument 'compressed' is ignored for the internal method")
  return(untar2(tarfile, files, list, exdir, restore_times))
}

## Further below
TAR <- tar
if (!nzchar(TAR) && .Platform$OS.type == "windows" && nzchar(Sys.which("tar.exe"))) 
  TAR <- "tar.exe"
if (!nzchar(TAR) || TAR == "internal") 
  return(untar2(tarfile, files, list, exdir))

In this case, the first untar2() call is called. That is: return(untar2(tarfile, files, list, exdir, restore_times)). The error message incomplete block on file is not really informative in this case because untar2() is not happy when there’s a space in the path to the file ^[Hopefully in the future Google will lead you to this blog post and you might avoid the rabbit hole I went through!].

We can get around this untar2() issue by uncompressing the tar.gz file ourselves in a path that has no spaces. For example, if we download DO.db_2.9.tar.gz to C:\R we can uncompress the tar.gz file with:

utils::untar('C:/R/DO.db_2.9.tar.gz')

Installation

Let’s proceed to installing the package.

> install.packages('C:/R/DO.db', repos = NULL, type = 'source', lib = 'C:/R/R-3.6.0/library')
* installing *source* package 'DO.db' ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
ARGUMENT 'Collado\AppData\Local\Temp\Rtmp8EQDjB\Rin2ef05088650f' __ignored__

Error: object 'ÿþ' not found
Execution halted

ERROR: lazy loading failed for package 'DO.db'
* removing 'C:/R/R-3.6.0/library/DO.db'

Warning message:
In install.packages("C:/R/DO.db", repos = NULL, type = "source",  :
  installation of package ‘C:/R/DO.db’ had non-zero exit status
>

Oh noes! It didn’t work 😖 What happened?

If you look closely, you’ll see that it says ARGUMENT 'Collado\AppData\Local\Temp\Rtmp8EQDjB\Rin2ef05088650f' __ignored__. Wait! Collado is the Lastname portion of the username! So we have another space issue ^[By the way, at this point I thought that the error was related to Error: object 'ÿþ' not found and maybe some encoding issues since the DO.db package has Chinese characters.]. That structure though looks very familiar, it’s from base::tempdir()!

> tempdir()

[1] "C:\\Users\\Leo Collado\\AppData\\Local\\Temp\\RtmpqiBJ53"

The help file for ?tempdir contained the clues to solving this issue.

By default, tmpdir will be the directory given by tempdir(). This will be a subdirectory of the per-session temporary directory found by the following rule when the R session is started. The environment variables TMPDIR, TMP and TEMP are checked in turn and the first found which points to a writable directory is used: if none succeeds ‘/tmp’ is used. The path should not contain spaces. Note that setting any of these environment variables in the R session has no effect on tempdir(): the per-session temporary directory is created before the interpreter is started.

We can set the TMPDIR environment variable which will be used by the R session spawned inside the installation of DO.db and… it works!

> Sys.setenv(TMPDIR = 'C:/R/tmp_leo')
> Sys.getenv('TMPDIR')
[1] "C:/R/tmp_leo"
>
>
> install.packages('C:/R/DO.db', repos = NULL, type = 'source', lib = 'C:/R/R-3.6.0/library')
* installing *source* package 'DO.db' ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package 'DO.db'
    finding HTML links ... done
    DOANCESTOR                              html 
    DOBASE                                  html 
    DOCHILDREN                              html 
    DOMAPCOUNTS                             html 
    DOOBSOLETE                              html 
    DOOFFSPRING                             html 
    DOPARENTS                               html 
    DOSYNONYM                               html  
    DOTERM                                  html 
    DOTerms-class                           html 
    DOTermsAnnDbBimap                       html 
    DO_dbconn                               html 
** building package indices
** testing if installed package can be loaded from temporary location
*** arch - i386
*** arch - x64
** testing if installed package can be loaded from final location
*** arch - i386
*** arch - x64
** testing if installed package keeps a record of temporary installation path
* DONE (DO.db)
Making 'packages.html' ... done

`clusterProfiler` installation

Now we can continue and install clusterProfiler, right?

> BiocManager::install('clusterProfiler', lib = 'C:/R/R-3.6.0/library')
Bioconductor version 3.9 (BiocManager 1.30.4), R 3.6.0 (2019-04-26)
Installing package(s) 'clusterProfiler'
also installing the dependencies ‘sys’, ‘formatR’, ‘askpass’, ‘farver’, ‘backports’, ‘zeallot’, ‘lambda.r’, ‘futile.options’, ‘curl’, ‘mime’, ‘openssl’, ‘hms’, ‘triebeard’, ‘tweenr’, ‘polyclip’, ‘RcppEigen’, ‘colorspace’, ‘utf8’, ‘vctrs’, ‘futile.logger’, ‘snow’, ‘data.table’, ‘fastmatch’, ‘stringr’, ‘httr’, ‘jsonlite’, ‘progress’, ‘urltools’, ‘xml2’, ‘gridGraphics’, ‘ggforce’, ‘ggrepel’, ‘viridis’, ‘labeling’, ‘munsell’, ‘R6’, ‘cli’, ‘crayon’, ‘fansi’, ‘pillar’, ‘BiocParallel’, ‘fgsea’, ‘reshape2’, ‘cowplot’, ‘europepmc’, ‘ggplotify’, ‘ggraph’, ‘ggridges’, ‘gridExtra’, ‘igraph’, ‘purrr’, ‘RColorBrewer’, ‘UpSetR’, ‘gtable’, ‘lazyeval’, ‘rlang’, ‘scales’, ‘tibble’, ‘viridisLite’, ‘withr’, ‘dplyr’, ‘glue’, ‘stringi’, ‘tidyselect’, ‘DOSE’, ‘enrichplot’, ‘ggplot2’, ‘GO.db’, ‘GOSemSim’, ‘plyr’, ‘qvalue’, ‘rvcheck’, ‘tidyr’

## Delete more output

The downloaded binary packages are in
        C:\Users\Leo Collado\AppData\Local\Temp\RtmpqiBJ53\downloaded_packages
installing the source packages ‘pillar’, ‘GO.db’


trying URL 'https://cloud.r-project.org/src/contrib/pillar_1.4.1.tar.gz'
Content type 'application/x-gzip' length 228572 bytes (223 KB)
downloaded 223 KB

trying URL 'https://bioconductor.org/packages/3.9/data/annotation/src/contrib/GO.db_3.8.2.tar.gz'
Content type 'application/x-gzip' length 31820866 bytes (30.3 MB)
downloaded 30.3 MB

Error in untar2(tarfile, files, list, exdir, restore_times) :
  incomplete block on file

Error in untar2(tarfile, files, list, exdir, restore_times) :
  incomplete block on file

The downloaded source packages are in
        ‘C:\Users\Leo Collado\AppData\Local\Temp\RtmpqiBJ53\downloaded_packages’

The issue is again that utils:::untar2() and thus utils::untar() does not like spaces in the paths. If we look at where the packages were downloaded more closely, we can see a space there at C:\Users\Leo Collado\AppData\Local\Temp\RtmpqiBJ53\downloaded_packages. If you check the help file for utils::install.packages() you’ll see that destdir controls this:

destdir

directory where downloaded packages are stored. If it is NULL (the default) a subdirectory downloaded_packages of the session temporary directory will be used (and the files will be deleted at the end of the session).

If we dig into utils::install.packages() we can see how this comes to play.

## Part of utils::install.packages()
if (is.null(destdir) && nonlocalrepos) {
  tmpd <- file.path(tempdir(), "downloaded_packages")
  if (!file.exists(tmpd) && !dir.create(tmpd)) 
    stop(gettextf("unable to create temporary directory %s", 
      sQuote(tmpd)), domain = NA)
}

Setting the environment variable TMPDIR doesn’t work here as the instructions for tempdir() specify ^[Didn’t stop me from trying hehe. I tried using usethis::edit_r_profile() and adding Sys.setenv(TMPDIR = 'C:/R/tmp_leo') but that didn’t work.] although I now see that you can edit the .Renviron file as instructed here.

In any case, if we specify a destdir without spaces we overide the need to control tempdir(), enable utils::untar() to work and we can finally install clusterProfiler 🎉.

> BiocManager::install('clusterProfiler', lib = 'C:/R/R-3.6.0/library', destdir = 'C:/R/dest_leo')
Bioconductor version 3.9 (BiocManager 1.30.4), R 3.6.0 (2019-04-26)
Installing package(s) 'clusterProfiler'
also installing the dependency ‘GO.db’

trying URL 'https://bioconductor.org/packages/3.9/bioc/bin/windows/contrib/3.6/clusterProfiler_3.12.0.zip'
Content type 'application/zip' length 623524 bytes (608 KB)
downloaded 608 KB

package ‘clusterProfiler’ successfully unpacked and MD5 sums checked
installing the source package ‘GO.db’

trying URL 'https://bioconductor.org/packages/3.9/data/annotation/src/contrib/GO.db_3.8.2.tar.gz'
Content type 'application/x-gzip' length 31820866 bytes (30.3 MB)
downloaded 30.3 MB

* installing *source* package 'GO.db' ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package 'GO.db'
    finding HTML links ... done
    GOBASE                                  html 
    GOBPANCESTOR                            html 
    GOBPCHILDREN                            html 
    GOBPOFFSPRING                           html 
    GOBPPARENTS                             html 
    GOCCANCESTOR                            html 
    GOCCCHILDREN                            html 
    GOCCOFFSPRING                           html 
    GOCCPARENTS                             html 
    GOMAPCOUNTS                             html 
    GOMFANCESTOR                            html 
    GOMFCHILDREN                            html 
    GOMFOFFSPRING                           html 
    GOMFPARENTS                             html 
    GOOBSOLETE                              html 
    GOSYNONYM                               html 
    GOTERM                                  html  
    GO_dbconn                               html 
** building package indices
** testing if installed package can be loaded from temporary location
*** arch - i386
*** arch - x64
** testing if installed package can be loaded from final location
*** arch - i386
*** arch - x64
** testing if installed package keeps a record of temporary installation path
* DONE (GO.db)
Making 'packages.html' ... done

Closing

All of the above seemed like too much. In addition, it seemed like BiocManager::install('hypeR', destdir = 'C:/R/dest_leo') was not working ^[I would need to test this more before reporting it properly to Bioconductor.]. I likely missed something here earlier today. So controlling utils::tempdir() seemed like the easiest solution such that the defaults of where a package gets downloaded, uncompressed, etc all worked. And the simplest solution we thought of was to create the C:\TEMP directory and update the Windows environment variables TMP and TEMP to point to that location. Then, the rest of the commands worked without having to specify lib or destdir or manually run utils::untar().

As a whole, remember to look for spaces in the error messages! This is specially relevant when you are having issues as a Microsoft Windows R user.

If you have other solutions for Microsoft Windows R users with usernames that have at least one space, please let us know in the comments! Thank you! 🙌🏽

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

[1] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

[3] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

The evolution of my academic career as seen through posters and talks thanks to hugo academic 4.1

Wed, 10 Apr 2019 00:00:00 +0000

The hugo-academic theme which powers my website is active and frequently updated. I don’t update my website that frequently anymore, but I recently found about many of their changes when I made the CDSB website.

We are delighted to share with you our new webpage at https://t.co/rNuiRlNixV with both English and Spanish support

Estamos encantados de compartirles nuestra nueva página web que viene en español e inglés

It'll replace/remplazará a https://t.co/SMtJbOM9KO #rstatsES #diversity pic.twitter.com/1SVg9rcqVf
— ComunidadBioInfo (@CDSBMexico) April 5, 2019

One of the new features that I liked quite a bit was the ability to have landing pages for each person in your team. I wanted to improve my website’s section describing the people I’ve mentored so I decided to update my personal website too. Once I started this process, I realized that talks and publications had drastically changed. You could now have an image per talk or publication, add tags to them and link them to projects. Furthermore, I noticed that I hadn’t uploaded all my posters nor the slides for all my talks. So this whole process of updating my website took quite a bit of time! All this new information is also reflected on my CV now.

The end result: alumni

First of all, I really like how the section describing the people I’ve mentored looks now. You can see the faces of the alumni and navigate to a page describing them in more detail.

For example, check Amy Peterson’s landing page which includes links to all her profiles I know about. I was also able to add some pictures of a key event with her: her MPH capstone final presentation.

My academic career

In this new version of my website you can also quickly take a look at my academic career by browsing through my talks and publications (papers, posters and a book chapter). The papers are fancier, but the posters and the talks tell you a more detailed story of how I’ve advanced through my career so far with in progress talks and posters showing some ideas, many of which we discareded in the end. For example, you can look at the poster I presented at BioC2010 which is when I met Rafa and Ingo from JHU Biostat. Through the posters and talks you can see how the derfinder project came to be and culminated in the publication describing the software, which was my main Ph.D. project. I like how all the talks and posters related to derfinder can easily be found through the project page. The talks now include my Ph.D. defense talk 🙌🏽✌🏽.

You can also see how the templates I used for making talks evolved through time starting with my joint undergrad project with Sur Herrera-Paredes. You can find some presentaitons made with Beamer, others with knitr (before rmarkdown existed), some with RStudio presentations, as well as different PowerPoint templates.

Overall, I’m very happy with my new website and I hope that you enjoy browsing through my academic career. I’ll use it to movitate others by showing them that we all start somewhere and it takes time and effort to grow.

Some code for udpating to hugo academic 4.1.0

If you are updating your hugo-academic website you should be prepared to spend a significant amount of time if you want to include all the details I included. It took me most of my Saturday, Monday afternoon, and several hours on Tuesday to update mine. Here’s some code I used in this process that might be helpful to you.

## I copied my lcolladotorsource/content/talk files to ~/Downloads/ugh-talks
## then I made a new directory named after the initial .md files
## I made this process easier when updating my publications
## (see further below)
p_ori <- '~/Downloads/ugh-talks/'
p_new <- '/users/lcollado/Dropbox/code/lcolladotorsource/content/talk'
f <- dir(p_ori)
ff <- gsub('.md', '', f)


## For testing
# i <- 1

for(i in seq_len(length(f))) {

    f_initial <- file.path(p_ori, f[i])
  f_new <- file.path(p_new, ff[i], 'index.md')
  
  initial <- readLines(f_initial)
  new <- readLines(f_new)
  
  ## For a set of tags present in the previous version of hugo-academic
  ## I was using and the latest one, I found the initial strings
  ## such that I could find and replace the text.
  ## I also made sure to delete that info on the previous version
  ## so I could inspect rapidly if there was any information
  ## on the old version that I hadn't transfered to the new version.
  ## This typically involved some custom urls whose syntax is now
  ## different.
  for(j in c('location', 'abstract ', 'event_url', 'url_video', 'url_slides', 'title', 'url_pdf', 'event ', 'math')) {
    ## for debugging:
    # print(j)
    patt <- paste0('^', j)
    cont <- initial[grep(patt, initial)]
    initial[grep(patt, initial)] <- ''
    stopifnot(length(grep(patt, new)) > 0)
    # print(length(cont))
    new[grep(patt, new)] <- cont
  }
  
  ## For times I had to do this manually
  ## since the names for these tags changed
  j <- 'time_start'
  patt <- paste0('^', j)
  cont <- initial[grep(patt, initial)]
  initial[grep(patt, initial)] <- ''
  new[grep('^date ', new)] <- gsub('time_start', 'date', cont)
  
  ## and the end time wasn't always there
  cont2 <- initial[grep('^time_end', initial)]
  cont2 <- gsub('time_end', 'date_end', cont2)
  if(length(cont2) == 0) cont2 <- gsub('time_start', 'date_end', cont) else initial[grep('^time_end', initial)] <- ""
  new[grep('^date_end ', new)] <- cont2
  
  ## Similarly, the short version of the abstract now has
  ## a different name
  j <- 'abstract_short'
  patt <- paste0('^', j)
  cont <- initial[grep(patt, initial)]
  initial[grep(patt, initial)] <- ''
  new[grep('^summary', new)] <- gsub('abstract_short', 'summary', cont)
  
  ## Replace the old and new files with the updated versions ^^
  writeLines(initial, f_initial)
  writeLines(new, f_new)
}




### Publications

## Again, I moved the original files elsewhere
## and I copied the exampleSite/content/publication/clothing-search
## directory

## I then edited the clothing-search/index.md file
## a little bit before copying it to all the new folders below

p_ori <- '~/Downloads/ugh/'
p_new <- '/users/lcollado/Dropbox/code/lcolladotorsource/content/publication'
f <- dir(p_ori)
ff <- gsub('.md', '', f)


for(i in seq_len(length(f))) {
  
  
  ## Create the new directories from R
  ## coping my modified template publication using the
  ## clothing-search example
  system(paste('cp -R', file.path(p_new, 'clothing-search'), file.path(p_new, ff[i])))
  
  
  
  f_initial <- file.path(p_ori, f[i])
  f_new <- file.path(p_new, ff[i], 'index.md')
  
  initial <- readLines(f_initial)
  new <- readLines(f_new)
  
  ## Update tags just like I did for the talks
  for(j in c('title ', 'date ', 'math ', 'publication ', 'abstract ', 'url_pdf', 'url_project', 'url_dataset', 'url_video', 'url_slides', 'url_code', 'authors ')) {
    #print(j)
    patt <- paste0('^', j)
    cont <- initial[grep(patt, initial)]
    initial[grep(patt, initial)] <- ''
    stopifnot(length(grep(patt, new)) > 0)
    #print(length(cont))
    new[grep(patt, new)] <- cont
  }
  
  writeLines(initial, f_initial)
  writeLines(new, f_new)

}

## I used imagemagick a few times to create the featured.jpg files
convert eurobioc2010-Bacterial-Collado.pdf[0] featured.jpg

## Other times I created those images by taking screenshots
## and then reducing the size to a max width/height of 1000 pixels
## to reduce the file sizes a bit

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

How to write academic documents with GoogleDocs

Tue, 02 Apr 2019 00:00:00 +0000

These past months I’ve been mostly working on one huge project which might be close to an end, hopefully! This project involves a massive manuscript with many supplementary figures and tables. Today we sent it out to other members in our team, and to celebrate, I’m now writing more 😅: though this is a blog post. I’m allowing myself to do so before I dive into the pile of tasks I haven’t completed¹. So I’m going to share with you the tools I’ve been using since 2018 or so for writing academic documents shared via Google Docs. You can use these tools for manuscripts², capstone projects³, and well, basically any document where you want to do any of the following:

Automatic figure/table numbering
Insert math equations
Cite the literature

Yes, you can do both with LaTeX and you can collaborate with others using Overleaf, but it’s really hard to convince others to use LaTeX in my experience.

Citations

There are many tools out there for you to organize the literature items you are reading or keeping tabs on. Some of them are Zotero and Mendeley, which you might have heard about. The one that I highly recommend for writing documents with Google Docs is F1000Workspace.

Get an account

First, you need to make an account. Worried about paying? Don’t worry, the accounts are free⁴!

I strongly recommend that you use your academic email here if you have one, because I believe that it grants you access for an unlimited free account. Your university might also give you free access.

Start a project

Once you have your account set up, start a shared project. I mean, private ones also work, but shared ones allow you to collaborate with others so that your teammembers can also update the citations in your document. For example, we have one called brainseq phase2.

Once you open your project page, at the top left you’ll see a big blue button called Import References. Click on it.

As you can see, F1000Workspace allows you to import references from many different sources. I typically import using identifiers, either a DOI or a PMID one. They also have a browser add-on that you can use to import references into your library when using websites such as PubMed.

Insert references into a Google Doc

On your Google Chrome browser, install the F1000Workspace Google Docs add-on available here. Next, open up your Google Doc and you’ll see that F1000 appears in your toolbar. If you click on it, the F1000 interface will open on the right sidebar.

That interface lets you link your Google Doc to a particular F1000Workspace project, which I recommend doing. You can then go to Insert citations and start searching your project citations. I typically search by name or by the identifier, which is particularly useful if I just added the reference to the project via the identifier on a separate browser tab.

Update your document’s bibliography

Lets say that you’ve added a few citations in your document and now want to format them appropriately. In your Google Doc, click on F1000, then navigate to the Format citations and bibliography section.

Before you click the big blue button that says Update citations and bibliography you’ll notice a dropdown menu that lets you choose your favorite citation style (or whichever the journal you want to send your manuscript requires).

The end!

Notes

Well, not really. You’ll likely keep adding many citations as you keep working on your document. One thing that I’ve noticed is that the F1000Workspace add-on has a bit of trouble under “suggesting” mode or when the citation was inserted as a suggestion. So I recommend that you accept the suggestion first, then use “editing” mode for updating your bibliography file. It’s always good to keep an eye on what the add-on is doing so you can notice anything weird and undo it with ctrl + z (cmd + z in macOS).

And hey, did you know that F1000Workspace also works with Microsoft Word?

For more details, check the FAQs.

Automatic figure numbering

Now that we have figured out citations in Google Docs, lets learn how to cross reference figures, tables, equations, and whatever else you want. This is something that LaTeX users are familiar with but that you can’t do out of the box in Google Docs or Microsoft Word (as far as I know). Luckily others have made add-ons that solve this problem. The one I use, and so do other 11,197 people as of today, is Cross Reference available from the Google Chrome Store for free.

This add-on allows the user to label equations, figures and tables and refer to them within the text. It now also allows users to create labels for any element. These elements are numbered automatically and references are updated to match. If their order changes, references update to match. If one is removed, references to it are highlighted in red in the text. The text and style of references and labels can be customised.

Insert labels and references as hyperlinks. Instead of a URL, add a code recognised by Cross Reference, then an underscore, then your choice of name.

Configuration

Once you install Cross Reference, you’ll see it listed under the Add-ons menu in your Google Doc.

You’ll see all the different types of elements that you have configured with Cross Reference. Some come out of the box, like Figures.

For every element you have to configure the following:

The code you will use for the label, here #figur.
The text that will be displayed before the number. In this case, Figure (yes, there’s a space there).
The style of the label; bold here.

Then the same thing for the reference. In this case, the code for the reference is #fig. The codes have to be at least 3 characters long for the references and 5 characters long for the labels.

In my documents I typically add configurations for supplementary figures and tables using:

type	code: label	code: reference	text
Supplementary Figure	`#sfigu`	`#sfi`	`Figure S`
Supplementary Table	`#stabl`	`#sta`	`Table S`
Supplementary File	`#supfi`	`#sup`	`Supplementary File`

Usage

Now that you have configured Cross Reference you can start using it. Lets say that I want to write the text We did many things (Figure S1) where Figure S1 links to my overview figure whose description starts Figure S1. Overview of my project. Since we are talking about the overview figure, lets use _overview as the unique identifier for this figure. As this is a supplementary figure, the label code is #sfigu and the reference code is #sfi. Meaning that we must write the label code once in the figure description using #sfigu_overview and we can reference to it as many times as we want using #sfi_overview.

The last tricky part is that you can write whatever you want, lets say hello, and then you need to create a link (shortcut is cmd + k in macOS) with the correct code (either the reference or the label one). So the text that you would write would be We did many things ([hello](#sfi_overview)) and later on when you describe the overview figure you need [whatever you want](#sfi_overview). Overview of my project where here I’m using the Markdown syntax for links: [text](link).

Once you’ve inserted the links both for the reference and the label, you can then go to Add-ons, navigate to Cross Reference and click Update document. Doing so will change the text you had initially filed in for the correct text. So it will look like this: We did many things ([Figure S1](#sfi_overview)) plus [Figure S1](#sfi_overview). Overview of my project. Here’s a real case example where my identifier for the first supplementary figure is _rna, thus the full reference code is #sfi_rna:

And you are done!

Notes

The number used for the item your reference depends on what order the reference codes are listed in the Google Doc. To check that the numbering order is correct (Table S1 appears in the text before Table S2, etc), I recommend opening your google document in two separate tabs. In one tab, you start reading your document from the top. If you encounter items out of order, then on tab two you can switch them around. That way you don’t have to scroll around and waste time, which is a more cumbersome problem as the document gets longer.

Overall, this process that takes a bit of time and can break due to a typo. So I highly recommend that you update your cross references as soon as you make a new one, so you can easily trace any typos and fix them easily. If you don’t, then it can become very hard to track down what went wrong.

Equations

Finally, lets say that you want to insert equations. You can insert some equations with Google Docs, but you might want more fine control. If you use LaTeX I recommend the Auto-LaTeX Equations add-on available from the Google Chrome Store for free.

This add-on lets you automatically convert every LaTeX equation in your document into beautiful images! Simply enclose your math equations within $$ … $$ and click the button in the sidebar, and all of your equations will be rendered in LaTeX!

It’s as simple as it sounds. What this add-on does is that it takes your LaTeX equation code, renders an image with the equation, and then inserts it back into your Google Doc. It also enables you to restore the LaTeX equation code so you can edit it if you find a typo.

Wrapping up

I hope that you’ll find this blog post / tutorial useful when writing your own academic documents. These tools have saved me so much time when writing academic documents in collaboration with others. I don’t want to imagine having to re-number all the references manually whenever we added each of the 48 supplementary figures, 17 supplementary tables, and 11 equations to the project we are about to complete 😱.

Acknowledgments

The authors of F1000Workspace, Cross Reference and Auto-LaTeX equations add-ons have made my life much easier. THANK YOU!!!

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Including answering some questions on the Bioconductor support forum, finishing another manuscript, etc. ↩︎
Like this one or this other one. ↩︎
Like my former student Amy Peterson did in 2018. ↩︎
For up to 3 projects. ↩︎

CDSBMexico: remember to apply for BioC2019 travel scholarships

Thu, 28 Feb 2019 00:00:00 +0000

This blog post was first published at the CDSBMexico website.

#CDSBMexico: remember to apply for BioC2019 travel scholarships!!

Due date is March 15thhttps://t.co/iegG0qQzwu

Let us help you! Here we give you some ideas 💡We can also give you feedback via Slack ✅#rstats #bioconductor @Bioconductor #bioc2019 #diversity #LatAm #rstatsES pic.twitter.com/EORg8d2Qxj
— ComunidadBioInfo (@CDSBMexico) March 1, 2019

About 10 months ago we announced our plans to start a new community of R/Bioconductor developers in Mexico and Latin America. The National Bioinformatics Node (NNB-UNAM in Spanish) has organized workshops in bioinformatics and R since 2006 and more frequently since 2010. Last year, 2018, there were five simultaneous one week workshops:

Introduction to R and RStudio
Exploratory data analysis (EDA) of biological data with R
EDA of RNA-seq data and differential expression studies
Genome assembly and annotation from high-throughput sequencing data
And our Latin American R/Bioconductor Developers Workshop

Overall it was a success as described in A recap of CDSB 2018: the start of a community. However, we are not done yet! 💪🏽As we have stated several times, our goal is to turn R users into developers and to increase the representation of Mexicans and Latin Americans at other R events.

BioC2019

The one R event that we followed as a guide last year is the Bioconductor yearly meeting, or BioC. This year, BioC2019 will be held during June 24 to 27 at New York City in the United States. If you are an academic the registration fee is 300 USD prior to May 24th. As of today (Feb 28th), a round trip from Mexico City on June 23 to 28th is about 320-410 USD¹. The accommodation options listed start at 118 USD plus tax a night. As you can see, these numbers add up fairly fast which is a limiting factor for scientists from Mexico (and Latin American in general) to attend BioC2019.

Travel scholarships

However, we want to remind you that there are travel scholarships available! And we, CDSB, want to help you! 🙌🏽

All the details about the travel scholarship and different types of submission proposals are available on the BioC2019 website. But let us summarize the process for you.

You have to apply for either a talk, poster or workshop.
You need a professional website, GitHub profile or your CV to be available online.
You need to answer two questions:

How have you participated in the Bioconductor project in the past?
What do you hope to get out of the conference?

For a talk or poster proposal you have a maximum of 300 words while workshop proposals involve filling out a template.

This is quite a bit of work, but the payoff is immense! You will compete for the opportunity to travel basically for free: free registration and hotel, plus $500 for travel expenses. This can be a career changing opportunity as it was the case for L. Collado-Torres (see his keynote slides) and others.

Get started on your scholarship application!

We know that many of us have a tendency to do things at the last minute. However, we hope 🙏🏽 that you start your application materials as soon as you can. Remember that the deadline is March 15th and no exceptions are made.

Website

If you don’t have a professional website, we recommend that you make one following Emily Zabor’s tutorial. This is the tutorial that one of your fellow CDSBMexico 2018 attendees used to create his website: joseaia.github.io. Alternatively, check out the Chromebook Data Science: intro to R course (available for free if you can’t afford it) which includes a lesson that walks you through this process.

Bioconductor project participation

Next, we know that each of you worked at least in one R package during CDSBMexico 2018 such as rGriffin (check the blog post). Maybe you even submitted your package to Bioconductor. At least, you met and interacted with longtime Bioconductor members such as Martin Morgan and Benilton Carvalho. Since then, you might have worked on other R packages, shiny applications, etc. You might have also answered questions on the Bioconductor support website, reported bugs or opened issues in existing packages, joined the developers mailing list, joined the Bioconductor Slack workspace, etc. Ultimately, you could potentially present a poster or give a talk about the package you worked on last summer, but you’ll need to coordinate with your team members and/or lab principal investigator.

Benefitting from BioC2019

As for what you hope to get out of the conference, well, we will remind you that BioC has different workshops that can help you advance your R skills and knowledge. Furthermore, there’ll be scientific talks from excellent bioinformatics researchers. Then, once you return from the conference, you can spread the knowledge you acquired in a variety of ways: implementing it on your work, teaching at your local university or research institute, participating in the Bioconductor community (Slack, the support website, contributing an R package, etc), presenting at a local R meetup, and participating in other forums. You can also benefit personally from attending the conference by networking with other users and developers that will attend it. From it, you might start new collaborations, find about graduate school programs, find potential postdoc advisors, among other opportunities. Hey, you might help us out with future CDSBMexico events!

CDSBMexico hands on help

If you need help directly, please get in touch with us via the CDSBMexico Slack workspace. If you either don’t remember the Slack workspace or you couldn’t attend CDSBMexico 2018, please get in touch with us via Twitter at twitter.com/CDSBMexico.

We understand that not many will be able to attend BioC2019, so keep an eye open for CDSBMexico 2019! We are hoping to announce our plans soon.

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Using Google flights for the search. ↩︎

Why do you like living where you live?

Tue, 12 Feb 2019 00:00:00 +0000

In the past months I’ve had a recurrent conversation with many people. This conversation is typically started with the question: why do you like living where you live? Some of them might be considering moving to the city I live in for work, some of them are thinking about leaving, some are happy here.

Ultimately, everyone is different and what makes some happy might not be for the rest. Some friends want to live in larger cities, others want different climates, others want to move in with their long distance relationship partners, etc. Back in 2015 I was finishing my Ph.D. and thinking about moving somewhere else in 2016. So I started to disengage from friends and the city itself. In the past couple of years, I’ve tried to change my attitude. Or well, adopt my parents’ attitude and follow their lead.

I frequently compared my current city versus my home city. This comparison used to be skewed in favor of my home city: my current city didn’t have the tacos I loved, or the quesadillas at the market, or my family, childhood friends, etc. At some point I started to point out in my head what my current city has that my home city doesn’t have or isn’t as accessible. One of them is a large open, free and largely secure park where you can ride a bike or play tennis like Patterson Park. So I pushed myself to go biking and play tennis in 2018. I also found a Mexican restaurant that feels like home. I made new friends that understood aspects of me that I didn’t show as much to others here, mostly due to cultural differences. And I started going more frequently to listen to the Baltimore Symphony Orchestra (BSO). So my advice for those who find their current city lacking versus their ideal city or hometown is to start looking for the activities that might be inaccessible elsewhere and that you enjoy. I really like the BSO, so I’ll get into more details about it.

First symphony

I really enjoy the BSO events and the story starts back in six grade in Madison, Wisconsin. That was my only year there and the school assigned me^[I don’t know if my parents chose for me.] to the chorus. That was new to me and I liked it quite a bit. At some point the chorus teacher offered some free tickets for a couple of students and their families to attend an event at a Madison symphony with a soprano solo. I don’t remember the name of the symphony nor the solo, but I was like, “sure, why not? Can I get the tickets?”. I knew that my family liked classical music and the teacher was happy that a student showed interest in the tickets. I had taken classical guitar lessons when I was younger because my dad loved it and we had several classical music CDs at home. I was blown away by the soprano solo despite my mom forcing me to dress up hehe. The second and last time the chorus teacher offered tickets, I was eager to get them, and we greatly enjoyed the event.

Classical music here and there

Across the years in Mexico, we would sometimes go to UNAM’s Sala Nezahualcóyotl. But it was always a bit of a trek: about 1.5 hr drive, early on the weekend, etc. It was maybe a bit too much and we would frequently get sleepy. I mostly listened to The Planets or Overture 1812 at home and particularly in family road trips. I remember one time crossing Aguascalientes where we had the volume super high during a portion where the symphony plays very low, and then boom, the symphony gets loud and my mom and brother were on the back of the car and complained quite a bit, but my dad and myself were really into it. So listening to the symphony reminds me about good times with my family.

BSO during grad school

The BSO offers student rush tickets for 10 USD the day of the event as well as a BSO student passport^[It’s now called BSO student select.] which is worth it if you go to 4 or more events in a season. Check the discount programs for more information. Through a combination of both I went several times to the BSO in my 5 years of graduate studies. But most importantly, I went to the season finales which tend to be on the first week of June. The season finale in 2012 was just after my masters comprehensive exam, and the one in 2013 was also just after my Ph.D. comprehensive exam. Both exams were brutal to study for and the BSO season finales were an incredible release of tension and energy for me. I felt many emotions during both finales and loved them dearly (they also featured choruses!). Over the years I went to many events and there was even a season when several of my grad school friends and I got the BSO student passports.

BSO since

After graduating, I got a BSO passport which is a great opportunity for those under 40 years old (nowadays it’s 99 USD) to attend as many events as you want in the season. I started to use the symphony as a way to work through my anxiety issues and mostly didn’t bother to invite friends. While that worked for me, in the 2018-2019 season I’ve been more proactive at inviting friends. I love it when a friend who hasn’t experienced the symphony much enjoys the show^[It’s not all classical music. Sometimes there are Broadway-like events, other times there are acrobats, movies with live soundtracks, or indie bands playing with the symphony.] or simply a reason to get out of their house and enjoy Baltimore a little bit more.

BSO opportunities and challenges

In recent years, I’ve also been donating to the BSO. I’m never a high donor or anything like that, but I like to contribute a little bit. I’m thankful for the wealthy donors that keep the BSO discount programs running and that I’ve personally benefited so much from. If you think about it, not long ago listening to a symphony was something only those in court did. Today it’s true that you can notice many wealthy patrons. But there are also opportunities for those with limited resources to attend and enjoy this world-class symphony.

However, there are also challenges. I’ve seen patrons complain about noises kids make: it’s hard to ask kids to stay quiet all the time. There are circumstances that magnify this: you can tell some kids have grown up surrounded by classical music while others haven’t. I think that a bit of tolerance can go a long way here to make sure newcomers feel welcomed.

One challenge that I think is very tricky is social media and phones. Many of my friends, myself included, like posting on social media. Taking videos and photos during most BSO events is strictly prohibited. I understand this from two reasons: first, the bright light of phones is disruptive^[Unless you reduce the brightness to a minimum.] and secondly, the symphony wants to try to motivate people to attend the shows in person. I either take photos before the show or at the very end^[It might help to emphasize this when annoucing that pictures and video during the event are prohibited.]. But it’s definitely a culture clash between what happens outside the symphony and what happens inside. We’ll see what time tells, but I do like that the BSO is trying to use Instagram more now. After all, social media in theory can help attract interest from others to attend.

Ultimately, the BSO faces strong financial challenges which you can read about in their updates section. They are doing many things, including a donation drive that ended yesterday for their anniversary. But as long as they continue their discount programs, this is a great opportunity that I encourage my friends to grab a hold on. It might even change your answer to the question: “do you like living in Baltimore?”, or well, play a role in it. It does play a role for me.

{width=600px}

That’s me with some friends at a BSO event in December 2018. We didn’t know then, but the BSO president took our picture for us 😅 Thanks again!

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

[1] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

[3] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

The power of tapping into your community for support

Sun, 03 Feb 2019 00:00:00 +0000

This week the owner of my favorite Mexican restaurant in Baltimore, Rosalyn Vera, got death and arson¹ threats. I could have been a bystander, but I tapped into my network and asked for help and she has received it. It’s been great to see the power of the community in action.

The backstory

So, I use R and Bioconductor for work and I get to witness the warmth and mostly friendly #rstats community where daily people ask for help and get it. Born from the R community is the RLadies organization that provides a platform for non-cis gendered males to share their knowledge and increase their visibility. In one of the RLadies Baltimore events, I met a Data Journalist from the Baltimore Sun, Christine Zhang who gave an excellent talk on how she uses R for her job: for example, election maps.

As you know, I’m Mexican. To get over my homesickness these past months I’ve been a frequent customer at a small Mexican restaurant in Baltimore called Cocina Luchadoras. I also frequent live shows from a band called Bad Hombres that plays covers of Spanish rock songs from the 80s and 90s, go to Latin karaoke nights organized by Root Raíces, among other things that bring me joy².

Initial threats and reactions

To warm up this week, I went to get some atole de arroz at Cocina Luchadoras twice. The second time was just about an hour after Rosalyn received death and arson threats on Thursday. I could tell that she was scared but she was also incredibly brave. The threats were because she has placed a sign on her restaurant to support the Donald fundraising campaign from Ilegal Mezcal that says “Donald, eres un pendejo” (Donald, you are stupid) . The person making the threats mentioned that immigrants are taking away jobs from US citizens. As a Hispanic customer, I got scared as well and became suspicious of whoever entered the restaurant.

So, what does one do? How would you react to a hate crime threat? Or well, any threat? Call the police. Do you comply with the request to take down the poster? Hire security? Close and move to a different location? I feel like situations like this put can you at odds between following your ideals and being pragmatic. In this case, Rosalyn had to decide whether to stand by her free speech rights or not.

In the United States of America you expect that the police and the institutions that protect the law will help you. Unlike Guatemala in the 1960s when the police/military tried to silence/kill my grandfather due to the health clinic he and my grandmother had where they would treat people from many backgrounds: he closed down, moved to Mexico, and had to be pragmatic and abandon some of his ideals. Or unlike Mexico in recent times, where there’s widespread insecurity due to the war on drugs, corruption, etc. For example, my father’s house was broken into, he got tied down and was punched until a few ribs broke: he called a Mexican military veteran for help.

So, here in the US, you call the police and wait for them to do their job. But you can also decide to ask for help from your community. So I offered Rosalyn my time and network, and she accepted. This was tricky as I told her, because I didn’t want to either give a platform for the person making the threats or to attract other people like him.

I asked Christine for help on whom to talk to at the Baltimore Sun, and she responded immediately. After some emails and knocking on a couple more doors, the Baltimore Sun published an article by Lillian E Reed with contributions by Thalia Juarez and Catherine Rentz. Catherine has published some great work on hate crimes in Maryland and they’ve made an interactive map to browse the data. From there, the word kept spreading and many started to show their support: Baltimore Reddit, local politicians, and many regular Cocina Luchadores customers.

The threats continue

I went again on Saturday and the place was packed and things seemed to be trending on the right direction. The police had called and were supposed to meet with Rosalyn later. Instead, they didn’t show up and the person making the threats called again…

From an allocation of limited resources, I understand that the Baltimore Police has many things to do and many people to keep safe. Catherine Rentz’ story on hate crimes in Maryland from 2016 and 2017 highlights that this type of crime is likely underreported and that there are very few convictions. Maybe this went through the mind of the policeman who did the report on Thursday: “threat phone call? No witnesses? Ok, not enough evidence… move on”.

I had a very hard time processing the no-show by the police though. Talking to others, we discussed how the law enforcement system is overworked (they are reducing the police officer shift length soon) and swamped. But hey, there’s a supportive community. So I asked for help again. “Who knows someone at the Baltimore Police?” That was the question and soon enough one friend replied, and then emailed a lieutenant. By the next morning (Sunday/today), the Baltimore Police chief was looking to call Rosalyn.

In the meantime, we let the Baltimore Sun know that the person had made the threat again. They quickly posted a new story where they report how Rosalyn first made sure to secure more witnesses this time and secondly told the person making the threats that she has only love for him and thanked him for the phone call. I think that she was really smart with both and she does mean it. In a chat with friends, one of them suggested bringing tacos as a peace offering to the person making the threats.

This second story by the Baltimore Sun got shared by Newsweek in their own story. A friend who is in Mexico right now told me he saw it on the top Apple News stories. Today, there was a security guard at Cocina Luchadoras, who smiled and opened the door for anyone coming in. So spreading the word, asking for help, knocking on doors, and the power of the community is very strong. Strong enough to get the police to pay more attention to Rosalyn’s report.

This is incredible! Typically, a #rstats/#RLadies request for help ends up with a friendly reply pointing to an existing R package or a suggestion on what to do. This time, we gamed the system. So yes, asking for help is challenging but can be worth it.

And surely everyone who’s involved in helping in this story can use it to request the help they need. For example, the Baltimore Sun’s hate crime reporting team can ask for more space and resources for their stories. Or the Baltimore and Maryland Police can ask for help from whoever controls their budget³, so they can get more resources to fight the increase in hate crimes, improve the reporting of hate crime threats, and reach out ot the Hispanic community in Baltimore. If they need a letter of support, they can get one from me if they want it.

For the rest of us, it’s a chance to try to live without fear. Fear is a powerful feeling and I’ve seen it in action. I experienced it first hand when everyone I knew stopped going out in Cuernavaca during an intense period of the war on drugs when military were deployed to the streets. It’s also a chance for us Hispanics in Baltimore to feel like we belong here, which is the same challenge many women and non-cis males have and hence the reason why RLadies is awesome.

And drop by to enjoy the awesome food and women art!

If you drop by in Baltimore, you should go visit Cocina Luchadoras! Their food is deliciously awesome. It starts with handmade tortillas. You use tortillas for many things including tacos. A good tortilla can boost the flavor in a taco, and the ones at Cocina Luchadoras are outstanding! Then there’s the filling which can range from pork (al pastor) to different types of vegetables. And while you wait, you get to experience a women’s empowerment museum with all the arts and crafts they have there about how women are really strong! You also get to meet many interesting patrons and make new friends if you wish to do so. Even if you are not in Baltimore, you should follow them on Instagram or Facebook for great art and food pictures (or Twitter, but they don’t post images there, just links to Instagram).

Acknowledgments

Thanks EpiRen!

Thanks Baltimore Sun staff and the Baltimore Police!

Thanks to the @baltimoresun for getting the word out about this hate crime threat against @CocinaLuchadora! 🇲🇽🇺🇸https://t.co/38oY5XDAK8

By @LillianEReed with contributions by @_ThaliaJuarez and @cdrentz

Thanks @christinezhang!

Thanks again @BaltimorePolice for investigating! https://t.co/rvXlHZBWj0
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) February 1, 2019

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Threats to burn down here business. ↩︎
Yes, thanks to Marie Kondo my clothes are neatly organized. ↩︎
The city council? The state congress? Mayor? Governor? I don’t know. ↩︎

Asking for help is challenging but is typically worth it

Mon, 12 Nov 2018 00:00:00 +0000

Recently I’ve been thinking on the subject of asking for help. In short, it’s hard to ask for help. It involves admitting to yourself that you can’t solve the problem alone, opening yourself up, hoping that another person will understand you and guide you in the right direction. Thus it can be painful if your request for help is misunderstood, met with criticism or ignored. Regardless of these obstacles, I think that the potential rewards make it worth it.

I mostly encounter the situation of asking for help in two scenarios. One is about work, mostly R programming. The other one is about personal issues. There are plenty more, like spelling, cultural references, word definitions and academic citations.

Thank you!

I’m always learning more English! Last night it was “yule” via @BenLangmead and others#DeepLearning? 🙃 pic.twitter.com/kX9JTnSBAn
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) November 10, 2018

PCR was invented by Kary Mullis in 1984. He won the Nobel Prize in chemistry for it in 1993. real-time PCR (qPCR) was developed about ten years later and originates with this paper: https://t.co/gpIOr3s7PG
— Lior Pachter (@lpachter) November 10, 2018

Programming context

In the context of R and Bioconductor, and programming in general, it can be scary to ask for help in public forums such as a GitHub issue page, the Bioconductor Support website, the RStudio Community website, etc. That’s because you might get a very short reply that can seem unhelpful if it involves terms you are not familiar with (though it might be a precise reply). There are also issues with online communities that I won’t get into. Over the years, I’ve tried to help those who will potentially answer my questions by:

Double checking the documentation¹.
Providing information to reproduce my problem: sometimes doing so makes me realize my mistake.
Trying for a while (>20 min?) to solve the problem by myself which involves googling quite a bit. However, sometimes I’m not aware of the correct search terms.
Avoiding sounding like I’m demanding help, asking for help immediately or requesting the same person for help lots of times in a short time window².

I’ve also tried to highlight my main question, though this can be hard when you are not sure what’s wrong.

When I get a short reply that seems unfriendly, I try to be appreciative of the time this other person took to reply to me. The terms they use in their reply can be important clues on what to search on Google next.

Language matters and I hate it when others make me feel dumb instead of guiding me to the right documentation/website, while others might dislike receiving tons of loose ended questions. Ultimately, this process requires patience on both sides and I highly value efforts by others that try to make it easier for both the person asking for help and the person responding. See for example:

Here are some my recent requests for help with replies by others:

I still make mistakes by not been clear enough sometimes, that’s why I typically say: “please let me know if you need more information” or something along those lines.

Personal context

Asking for help in a programming context is hard because you are leaving your questions public (most of the time): that way you can reach multiple people that can potentially answer your question, and the people answering the question want them to be public with the hope that the answers will help others. Asking for help in personal context is different, since it tends to be in private.

This might just be me, but I’ve had several situations in my life where I’ve opened up and asked a group of individuals for help. For example, I recently started a conversation with several friends about ways to limit alcohol consumption. It was scary at first to bring this topic up, but the friends I talked to about this subject had also been thinking about it and it lead to some great conversations. I have other examples listed in my timeline³ on occasions where others have helped me with career advice.

One story that I want to share now is about grad school. I was a first year student and I had an accident one Friday in February. I knew immediately that I needed a surgery to repair my shoulder. I already had experience with orthopedic surgeons in Mexico whose quality was widely variable: it took my family years to find one we could trust for my first surgery. In my first year at a new city and country, I didn’t have my family’s network support. So I reached out via email to the faculty that I had met in person with the hopes that one of them knew a good orthopedic surgeon. This was a personal help request, so I didn’t use the Department wide mailing list. The response was excellent, and not only did some faculty help me find an excellent orthopedic surgeon, but also get an appointment with him for Monday. After my appointment on Monday, I replied to the same set of faculty thanking them for their support.

Later that week, one faculty dropped by my office to reprimand me for emailing all the faculty. I assume that this faculty thought I had emailed the whole Department. This situation shocked me and made me feel awful. My mind started racing and I wanted to clarify the situation with the faculty member. It was taking a lot of my internal courage to approach this faculty member and a second faculty member noticed that I wasn’t well. This second faculty member listened to me⁴ and later invited me to have dinner with his family.

A Department staff member also helped me during these weeks and pointed me to the Johns Hopkins Student Assistance Program where I got help (2 or 3 sessions). Getting professional help was great; some things I remember were that they explained to me that I was mourning since injuring my arm was like losing a piece of me (given my injury history) and they also explained potential cultural differences that lead to the situation with the first faculty member. Once I calmed down, I did email the first faculty to clarify what had happened.

This was a rough situation and probably my toughest moment in grad school, or at least among the top ones. I also value it as a learning moment and it fueled me and continues to do so in some ways. For example, it motivated me to organize Cultural Mixer events (see the ad for the third event), with the hope that they would help us understand each other better⁵, as well as creating the Biostat Social mailing list (now defunct but replaced by a Department Slack). I also think that the reactions to my request for help were overall positive. The balance for sure is positive although there was a bit of negative in the mix. I wish others would just get positive reactions with zero negative ones, but there are no guarantees.

Closing thoughts

With this post I’m hoping to invite others to reflect on situations that involve asking for help. I also hope that others break their internal barriers and ask for help when needed. Echoing words of others, asking for help takes courage. Given the potential rewards, I encourage you to ask for help from your family, friends and colleagues depending on the situation. Simply knowing the term(s) for what you are searching for in code or for what you are feeling (for example, imposter syndrome) can help a lot. Also being directed to professionals, like JHSAP in my shoulder story, and the proper documentation manual, like the R admin devel manual, can propel you in the right direction.

Best of luck walking the uphill path of asking for help!

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Sometimes I simply don’t understand the documentation, or I’m not aware of the best documentation for my problem. ↩︎
Doing any of these typically won’t get you anywhere. The person(s) responding to your help requests is(are) also busy! ↩︎
Explained in more detail in this post. ↩︎
Or tried to because I couldn’t contain the tears. ↩︎
Among other great reasons, I’ll write a blog post about them sometime soon. ↩︎

A knot of threads: from CSHL to LCG-UNAM to Aldo Barrientos to diversity scholarship opportunities

Tue, 06 Nov 2018 00:00:00 +0000

I can’t tell you how many times I’ve started to write this post in my mind since May 2018. Today I’m finally typing it on the computer. This will be a rather long post that ties in several threads. I’ll talk about Cold Spring Harbor’s Biology of Genomes conference and its relationship to my undergrad in Mexico. I’ll also introduce you to Aldo Barrientos (1987-2011) who was was my undergrad classmate. Then I’ll tell you about myself and how regardless of your situation, privileged or not, you should ask for help and get to know the people around you. Finally, I want to highlight that many other Mexican researchers are doing great work and would love to be invited to talk about their work. Or simply, the opportunity to compete for scholarships and grants.

CSHL and LCG-UNAM

This past May I was selected to give a talk at Biology of Genomes 2018 (BOG) at Cold Spring Harbor Laboratory (CSHL). This was my first time being there and I had plenty of time to reflect before my talk¹. For me, attending BOG was like a dream come true so I was blown away when I was selected to present at BOG.

I first heard about the BOG conference when I was studying my undergrad in Mexico at LCG-UNAM. My class has a history with the conference too. Basically, part of the class was meeting to discuss what optional courses to create for our fourth year, and well, some miscommunication happened among us. This created a rift in the group and was one of my motivations for focusing on communication as well as giving back. The former lead to me being elected as class president and the latter lead to me volunteering to teach R courses later on my undergrad².

As BOG18 went underway, some of my fellow junior presenters started their talks by mentioning why CSHL was special to them. Following their cue, I wrote some notes just before my presentation.

I really wanted to give a shoutout to all my LCG-UNAM classmates. I also thought of Aldo Barrientos and how it could have been him at BOG. This got me nervous, but I really wanted to do it. I also wanted this to be short and professional. You can see how it turned out for yourself³.

Aldo Barrientos

It’s time for me to tell you more about Aldo Barrientos, my LCG-UNAM classmate. I won’t attempt to tell you his whole story since I think that others could write a much more detailed one that myself. I hope they do and if they are made public, I’ll share them from this blog post.

Aldo came from a low income family in Mexico: I believe that his father was a construction worker⁴. He was smart and advanced through school successfully. He took the entry exam for LCG-UNAM like many people did in 2005. Eventually only around 10-15 percent of the applicants were admitted to this undergrad program. The school was free (25 Mexican cents per year), but it involved relocating to a new city. UNAM has some scholarship programs, but they are not enough. Aldo found a way like he always did. For example, he sold candy and juice to his classmates during lunch break. For reasons that I don’t remember, Aldo didn’t complete his undergrad in 2009 but was about to in 2011.

I remember Aldo as a very joyful person who was always smiling and joking with the rest of us. He had the best dancing skills of anyone at my undergrad. He proved them time and time again, for example winning a dance contest at a party in my house and wearing the Pumas UNAM puma costume in our graduation party and dancing with everyone.

Image source

His path was cut abruptly in 2011. He was stabbed multiple times while waiting for someone. We all knew about this. It’s still unclear to me how everything happened in the following months. But I remember being told that he tried seeing a doctor before going back to his home. One day in 2011, we all started to learn about how grave his situation had turned. I didn’t know what to do⁵ and I talked to David Romero, director (dean?) of CCG-UNAM back then. He guided me and others and we were trying to coordinate ways to help Aldo, but by the time we got to his home it was too late. I regret that we weren’t able to help Aldo earlier.

I still have some emails from 2011 when we were talking about doing something to honour him. We never did as a class, though LCG-UNAM did acknowledge him. Maybe one day we will do an event to remember him.

Aldo faced more challenges that myself and other classmates. Yet I believe that he would have kept solving them one by one; he would have gotten his undergrad degree and had he chosen to stay in academia, then present his research findings at conferences like BOG.

Privileged yes, entitled no

Thinking about Aldo made reflect also on who has been around in my own path, who has helped me and how grateful I am to them. Undoubtedly I’ve had an easier path than others including Aldo. That is, I’ve been privileged.

Still, even with my privileged position it has been a hard fought battle to get to where I am now. I think that a mistake others do is that they assume that they are entitled to something given their privilege instead of working for that thing. This can apply to visas and the like where you still have to apply for them in time.

Along my way I have hesitated many times what to do and have sought guidance from others. Also along my way, I’ve tried to get to know others in my surroundings. Now that I look back I realize that many people were at my position or in junior positions have kept working towards their career goals and advanced in them. So, if you can, get to know others and learn from them. They could be the future Full Professors with Tenure, future Senior Software Engineers or other titles you look up to.

Here’s an incomplete version of my timeline (PDF version). I think that it’s pretty hard to list everyone who has helped you along the way. For example, I’m not mentioning my Ph.D. classmates from whom I learned many things. But it will give you a more detailed picture of the amount of people that have helped me.

Invite LCG-UNAM alumni

Like I said at the beginning of my BOG18 talk, I’m not sure if I’m the first LCG-UNAM alumni to present at BOG but I know for sure that I won’t be the last one. That’s because I’m really proud of my classmates and all the great work they are doing. It’s also easy to go beyond my class to other LCG-UNAM alumni and then even go beyond and extend this to Mexican researchers.

Taken from my timeline, here are a few that I mentioned there (links are to their Twitter profiles):

Maria Gutierrez-Arcelus: LCG-UNAM; Ph.D. at University of Geneva Medical School; Postdoc Harvard University
Mariana Gómez-Schiavon: Ph.D. Duke University; postdoc UCSF
Sur Herrera-Paredes: Ph.D. University of North Carolina; postdoc Standford
Fernando Rabanal Mor: Ph.D. Gregor Mendel Institute; postdoc MPI for Developmental Biology
Carlos Vargas: Winter Genomics, MS and Ph.D. Universitat de Valencia
Mariana Reyes: MS and Ph.D. Universitat de Valencia
Atahualpa Castillo: Ph.D. and postdoc University of Bath
Jimena Mónzon (Ph.D. Univeristy of Bath; Oxford’s Parkinson Disease Centre Career Development Fellow)
Alejandra Zayas: Ph.D. IBT-UNAM
Adriana Verenisse Gonzalez Sandoval: Ph.D. University of Bath; postdoc Stanford
Daniela Robles: Ph.D. Wellcome Trust Sanger Institute; Faculty LIIGH-UNAM
Luis Bolaños: Ph.D. CCG-UNAM; postdoc Oregon State University
Rodrigo García: MS and Ph.D. Universitat de Valencia
Renan Escalante-Chong: Ph.D. Harvard postdoc MIT; Immuneering Corporation
Gianella Garcia Hughes: Ph.D. UT Southwestern
Adrian Cantú: LCG-UNAM; MS IBT-UNAM; INMEGEN; Ph.D. San Diego State University
Ana Georgina (Yina) Cobian: MS IBT-UNAM, Ph.D. San Diego State University
Fernando Riveros-McKay Aguilera: Winter Genomics, Ph.D. Wellcome Trust Sanger Institute
Martín Del Castillo Velasco Herrera: Winter Genomics, Ph.D. Wellcome Trust Sanger Institute
Alejandra Medina-Rivera: LCG-UNAM; Ph.D. CCG-UNAM; postdoc SickKids Toronto; Faculty LIIGH-UNAM
Alejandro Reyes: LCG-UNAM; Ph.D. EMBL; postdoc Harvard
Víctor Moreno Mayar: LCG-UNAM; Ph.D. and postdoc Natural History Museum of Denmark
José Reyes: LCG-UNAM, Ph.D. Harvard

For the full list of LCG-UNAM alumni, check http://www.lcg.unam.mx/es/titulados. There are 257 listed as of October 25, 2018.

Everyone has their own story of breaking barriers and overcoming challenges. All of them would love the opportunity to present their work and to compete for funds to travel to conferences and do so.

It is and for example Mexican grants cannot be used for paying memberships so we cannot easily get the discounts.
— Alejandra Medina-Rivera (@AleMedinaRivera) October 17, 2018

To close off, here are some inspiring or thought provoking tweets:

"Along with a healthy dose of luck, the key attributes needed to produce a worthy PhD thesis are a readiness to accept failure; resilience; persistence; ...and a willingness to commit to very hard work — together with curiosity and a passion for research." https://t.co/o62DODM3iZ
— Mariana Gómez-Schiavon (@mgschiavon) October 28, 2018

When you're early-stage in your career, reading stories of people who have been successful is invaluable.

My #rstats hero is @JennyBryan, and she's shared her story in 2 interviews:
-@rOpenSci profile by @kellrstats: https://t.co/bz9Jj4UJqG
-@statschat: https://t.co/UFWfgIg2qa
— We are R-Ladies (@WeAreRLadies) October 24, 2018

I was told by a PhD thesis committee member that if I returned to Mexico after a postdoc I would "fade into mediocrity". However, I did return to my home country, started my own lab, and are now well funded to investigate the genetics and genomics of cancer in Mexican patients. https://t.co/9inVPvYkPj
— Daniela Robles @daniela-oaks.bsky.social (@daniela_oaks) July 18, 2018

Got a bad news myself today but then I got an AWESOME one from my mom! She got a 9/10 in her chemistry class, her last class to complete her high school!!! I’m so proud of her & how resilient she’s been! My brother & I couldn’t be happier, prouder & dying to hug her! #nevergiveup
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) May 31, 2018

In Mexico, Day of the Dead was just last week (November 2nd) and Aldo’s birthday would have been a few weeks ago on October 27th. It was time to finally write this blog post. I hope that you take something positive from it.

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

Code for the dates:

## R code for making the dates.pdf file
## which I then imported into Adobe Illustrator for making
## the timeline.png
x <- paste0(1987:2018, '-', c(paste0('0', 1:9), 10:12), '-01')

plot(y = seq_len(length(x)), x = as.Date(x), las = 2, ylab = '', xlab = '', xaxt= 'n')
axis(1, at = as.Date(x), labels = months(as.Date(x)))


library('ggplot2')
library('scales')
df <- data.frame(y = 0, date = as.Date(x))


pdf('dates.pdf', height = 40,  useDingbats = FALSE)
ggplot(df, aes(y, date)) + 
  geom_line() + 
  scale_y_date(limits = c(as.Date("1987-07-01"), as.Date("2018-05-12")), 
               breaks = date_breaks('months'),
               date_labels = "%Y-%m", expand=c(0,30)) +   theme(axis.text.y=element_text(angle=-180))
dev.off()

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  colorspace      2.1-0      2023-01-23 [1] CRAN (R 4.3.0)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  dplyr           1.1.2      2023-04-20 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fansi           1.0.4      2023-01-22 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  ggplot2       * 3.4.2      2023-04-03 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  gtable          0.3.3      2023-03-21 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  munsell         0.5.0      2018-06-12 [1] CRAN (R 4.3.0)
##  pillar          1.9.0      2023-03-22 [1] CRAN (R 4.3.0)
##  pkgconfig       2.0.3      2019-09-22 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  scales        * 1.2.1      2022-08-20 [1] CRAN (R 4.3.0)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  tibble          3.2.1      2023-03-20 [1] CRAN (R 4.3.0)
##  tidyselect      1.2.0      2022-10-10 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  utf8            1.2.3      2023-01-31 [1] CRAN (R 4.3.0)
##  vctrs           0.6.3      2023-06-14 [1] CRAN (R 4.3.0)
##  withr           2.5.0      2022-03-03 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Reactions to the blog post

My good old friend Leo sharing very interesting experiences of his path through academic life, public university system in #Mexico, and remembering a very special person who taught us a ton of life lessons. https://t.co/nHAiob30jn
— Luis Bolaños (@lbolanos68) November 7, 2018

I'm reading this with tears in my eyes and joy in my heart. I'm so grateful and proud of my friend Leo. I'm so proud of all my classmates! Love you all! Love you Aldo <3 https://t.co/ifOK9ASPmy
— Mariana ReyesPrieto. (@mreyespri) November 7, 2018

Beautiful and inspiring!! Thanks Leo for writing this, for honoring Aldo and being the voice of many (including myself) who feel lucky and grateful for being a LCG alumni! https://t.co/hDyKKThj8B
— paleogenomics (@paleogenomics) November 8, 2018

Mine was the very last one of the conference. ↩︎
Which then lead to my current career. See more at my keynote for the “Latin American R/BioConductor Developers Workshop 2018”. ↩︎
My scientific talk was about the BrainSeq Phase II project by the Lieber Institute for Brain Development. It’s now available as a bioRxiv pre-print: 426213. It took a while to finish this project, which contributed to my delay in making this blog post. ↩︎
Previous versions of my post incorrectly said that he was a farmer. ↩︎
I believe that others asked me since I was or had been the class representative. ↩︎

Steps for writing a Twitter summary conference blog post

Tue, 23 Oct 2018 00:00:00 +0000

I recently wrote several blog posts with many tweets¹ as a way of taking notes during the American Society of Human Genetics 2018 conference. A few friends asked me how I did this so fast. The process can be summarized into the following main steps.

Save links of tweets you want to highlight in your post.
Use a hugo-powered blog to obtain the code for embedding tweets easily².
Proofread, edit and post.

My steps detailed

Here’s the actual process I used.

Identify the conference Twitter hashtag. In my recent case, that was ASHG18.
Create a Google doc or some other simple text file you can access easily from your phone and laptop.
While in a presentation, check the latest tweets for the given hashtag. ASGH18 example.

Copy the link of any tweet you find interesting. If there are none for the session you are in, consider writing some tweets yourself with the conference hashtag!

Paste the link into your Google doc.

Once the day is over, copy-paste the contents of your Google doc into a text editor that has the ability to find and replace using regular expressions. For example, TextMate.

Find and replace to create the syntax for embedding tweets. I’ll cover more of this in detail in the next section.

Find:

https://twitter.com/.*/status/

and replace by:

\n`r blogdown::shortcode("tweet", "

Then find:

\?s=[0-9]*

and replace by:

")`\n

Our file now looks like this:

Add any comments you wish, format the headings and proofread.
Make it public and tweet about your post with the conference hashtag!

And you are done! 🙌🏽🎉

Quick embedding

Note that my blog is powered by blogdown (Xie, Hill, and Thomas, 2017) which has the function blogdown::shortcode() for embedding tweets. For example, blogdown::shortcode("tweet", user="lcolladotor", id="1054380320533950464") generates the hugo syntax code which then hugo renders into the tweet itself.

I just completed my #ASHG18 survey @GeneticsSociety. Took a few minutes and most of my feedback wouldn't stand out. Except that I did use a box to suggest having a webinar with role playing scenarios that exemplify how the code of conduct can be used to diffuse harassment @jsdron pic.twitter.com/AKZdnGCr25
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 22, 2018

As an alternative, we can use Twitter itself by copy-pasting the code for embedding the tweet directly from the Twitter app/website. On the top right side menu on the tweet itself, select Embed Tweet

then copy-paste the HTML code into your post.

Here’s the code:

<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">I just completed my <a href="https://twitter.com/hashtag/ASHG18?src=hash&amp;ref_src=twsrc%5Etfw">#ASHG18</a> survey <a href="https://twitter.com/GeneticsSociety?ref_src=twsrc%5Etfw">@GeneticsSociety</a>. Took a few minutes and most of my feedback wouldn&#39;t stand out. Except that I did use a box to suggest having a webinar with role playing scenarios that exemplify how the code of conduct can be used to diffuse harassment <a href="https://twitter.com/jsdron?ref_src=twsrc%5Etfw">@jsdron</a> <a href="https://t.co/AKZdnGCr25">pic.twitter.com/AKZdnGCr25</a></p>&mdash; 🇲🇽 Dr. Leonardo Collado-Torres (@fellgernon) <a href="https://twitter.com/lcolladotor/status/1054380320533950464?ref_src=twsrc%5Etfw">October 22, 2018</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

and here is how it looks:

I just completed my \#ASHG18 survey (**GeneticsSociety?**). Took a few minutes and most of my feedback wouldn't stand out. Except that I did use a box to suggest having a webinar with role playing scenarios that exemplify how the code of conduct can be used to diffuse harassment (**jsdron?**) pic.twitter.com/AKZdnGCr25
— 🇲🇽 Dr. Leonardo Collado-Torres ((**fellgernon?**)) October 22, 2018

There might be other ways of doing this that are specific to your blog platform.

I also imagine that it should be possible to bookmark the tweets on Twitter and then use an R package to extract your bookmarked tweets from the past 24 hours in order to simplify the process. Maybe this can be done with rtweet although I haven’t dived into it.

New to `blogdown`?

Check:

Also, ask around. You might belong to a group or know someone that is willing to host a guest post in their blogdown blog. For example:

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

ASHG18 day 1 post, day 2, day 3, day 4 and day 5. ↩︎
Or any other tool that can get you the embedding code, including the Twitter page itself. ↩︎

ASHG18 tweet summary day 5

Sat, 20 Oct 2018 00:00:00 +0000

Continuing from my ASHG18 day 1 post, day 2, day 3 and day4 here’s my list of tweets from day 5.

6C 9:15 am

Jane Loveland

Wish that gene annotation was consistent across databases? Jane Loveland is speaking about the new #MANE project which aims to converge transcript annotation between @GencodeGenes and @NCBI RefSeq in 30 min 09:15-09:30, Room 6C, Talk 302 #ASHG18
— Ensembl (@ensembl) October 20, 2018

Jane Loveland: there are two comprehensive transcript annotations, RefSeq and Gencode. Clinical labs use RefSeq, basically everyone else (e.g. gnomAD, GTEx) uses Gencode. Describes the MANE project to unify these annotations. #ASHG18
— Daniel MacArthur (@dgmacarthur) October 20, 2018

Jane Loveland

Annotation project leader. #MANE project is based on GRCh38

Matched Annotation NCBI and Ensembl (I think)

Looking for “longest strong” transcripts

Want stable yet updatable pipeline

Many challenges!

#ASHG18 @ensembl @NCBI
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 20, 2018

Jane Loveland

Goals for Phase 2: update remaining 47% genes

Challenges: missing data for some genes!

No cage
No polyA
No … (missed it)

A to audience Q: Once they select a transcript, they’ll keep it. #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 20, 2018

.@princyparsana asked Jane Loveland about overlapping genes. They are addressing this issue! #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 20, 2018

6B 10 am

Genevieve Stein-O’Brien

Genevieve Stein-O'Brien describing her package CoGAPS - in Bioconductor - to learn both discrete and continuous axes in single cell data. #ASHG18
— Nick Banovich (@NeBanovich) October 20, 2018

Yu Hu

Yu Hu

On using single cell RNA-seq data for detecting differential splicing events

Presented SCATS and compared to MISO derived results

Can we recover cluster pattern?
Check sensitivity

SCATS is his thesis projecthttps://t.co/1c52ulw6e3 #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 20, 2018

Mingyao Li and Nancy Zhang were his advisors.

Hierarchical model with normal + (missed it) + log + poisson

20 BC 10:30 am

Stephanie Kravitz

@snkravitz Our textbook model of gene expression is that each parental allele is expressed equally. That is, biallelic expression. However, both genetic and epigenetic mechanisms can lead to monoallelic expression; one or mostly one allele is expressed instead of both.
#ASHG18
— Aaron Quinlan (@aaronquinlan) October 20, 2018

@snkravitz Her goal is to identify autosomal genes that exhibit monoallelic expression.#ASHG18
— Aaron Quinlan (@aaronquinlan) October 20, 2018

@snkravitz Low madRD indicates biallelic expression. High madRD indicates monoallelic expression. madRD robustly distinguishes genes on the X from autosomal.#ASHG18
— Aaron Quinlan (@aaronquinlan) October 20, 2018

@snkravitz Goal is to build a map of tissue-specific genes that exhibit monoallelic expression. Distinguish genetic (e.g., eQTL) and epigenetic effects.
How does allelic expression modulate the penetrance of disease alleles?#ASHG18
— Aaron Quinlan (@aaronquinlan) October 20, 2018

Li Chen

Li Chen

Looking at eQTL associations with SNVs.

Developed TIVAN pipeline for fine mapping eQTL SNVs. Compared against other methods.

Used GTEx v7 data + eQTL data from other studies (missed the full list)#ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 20, 2018

Li Chen

Tested model using:

* Cross validation
* Leave-One-Chromosome-Out (LOCO strategy); means crazy 😜 in Spanish 🇪🇸 🇲🇽 #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 20, 2018

Li Chen handled very well a question on “have you compared to X new paper?” with “great question, paper X didn’t come up in our literature review but we’kk look into it”

Li Chen also gave a shoutout to his trainee in computer science who did most of the work #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 20, 2018

Nicholas Bogard

Nicholas Bogard

Last talk of the conference! 👋🏽

Using a Deep Neural Net with lots of data. 🌊

Looked a PolyA signal (PAS) sequence.

Built a APA reporter library (generated over 3 million unique PAS)

Built APARENT: APA regression net with computational collaborator#ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 20, 2018

Nicholas Bogard

Looked at predicted results from APARENT and proximal/distal PAS (pPAS/dPAS). SNVs, ClinVar, CES (a short sequence) mutation impact

Also looked at the distribution of predicted cuts#ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 20, 2018

Other

Plewcczynski #ASHG18 Now collab with JAX; 4D Nucleome NIH project described here https://t.co/CP6ZmPR1ta Sequence infl chromatin structure
— Dale Yuzuki (@DaleYuzuki) October 20, 2018

.@tuuliel discussed v8 of GTEx release which should be out this coming winter. ~17K samples, 54 tissues, 838 donors. 24K eQTLs but only <200 trans-eQTLs. #ASHG18 #ASHG2018
— Jason Miller (@JEMgenes) October 20, 2018

DW: PsychENCODE and other consortia genomics data integrated to understand functional genomics of brain disorders. Brain transcriptome and epigenome, single-cell data, QTL data. Build a deep neural network to predict disorders #ASHG18
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 20, 2018

.@taibo_li introducing biological networks as a data structure in his single cell co-expression network talk. Tool (with cute cat mascot): https://t.co/fX1aF1FJxQ #ASHG18
— Avery Davis Bell, PhD (@averydavisbell) October 20, 2018

Diversity

As I was leaving, I said that I was off to visit the poster of an old PhD student in my lab. He asked, “Your lab?” and I clarified that no, I meant the lab I worked in. He then responded, “That makes sense - you’re too hot to be a PI!” 2/5
— Jacqueline Dron, PhD (@jsdron) October 20, 2018

If you want, you can still do something now. Like find his name from the app map/poster number, double check on google, then report him. See https://t.co/Lwb70wbqLo for info on how to report this person.

Though I can understand if you don’t want to relieve this exp. #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 20, 2018

We are so very sorry to hear of this, and encourage you to report violations of our code of conduct through our online system: https://t.co/4j0ChnHbXW Reports will be taken seriously and addressed in a timely manner.
— ASHG (@GeneticsSociety) October 20, 2018

SOMEONE MIND CLUIN' ME IN ON WHAT A 'BUZZ WORD' IS?? SOUNDS LIKE BUZZARD, BUT I KNOW THEM ARE ONLY RESERVED FER THE CARCASSES OF NO GOOD SCOUNDRELS THAT GO 'ROUND HARASSIN' WOMEN TRAINEES AT THEIR POSTERS. #ASHG18 https://t.co/dGHtEFNm1L
— The Sheriff of ASHG (@genome_sheriff) October 20, 2018

Take home message of #ASHG18 is we need more #GWAS for different populations!
— Şeyma Katrinli (@SeymaKatrinli) October 20, 2018

"Genomics has a diversity problem" - genetic #diversity in #GWAS peaked and stagnated in 2014, more diverse GWAS and new methods are needed to avoid potential for #polygenicriskscores to increase #healthdisparities @GeneticsSociety #ASHG18 pic.twitter.com/bWy34u0qp1
— Terence Wong (@terencecwong) October 20, 2018

Misc

Am I the only person that prefers ‘protein truncating variant’ instead of ‘Loss of function’? So many great talks/examples at #ASHG18 where LOF is not an accurate description but LOF still seems to be the default name.
— Janson white (@JansonWhite) October 20, 2018

LOL I'm not typing that equation #ASHG18
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 20, 2018

It is so fun seeing big, confident smiles on the faces of trainees after giving a great talk. The best.#ASHG18
— Aaron Quinlan (@aaronquinlan) October 20, 2018

From Ambrose Wonkam on genetic medicine research in Africa, to @genetisaur & @barbara_dutty on probs calculating PRS with euro data in non-euro’s, and Azeez Butali’s cleft palate GWAS, one main #ashg18 takeway is the value of genomic research in African populations @AfSHGenetics
— Sarah Spendlove (@spendlove_sarah) October 20, 2018

How can I take a computational talk serious if the first slide has a 3D pie chart? #ASHG18
— Wouter De Coster (@wouter_decoster) October 20, 2018

#ASHG18 Nathan Abell says wet lab + dry lab = moist lab.
— Testing enthusiast (@apicoplast) October 20, 2018

#ASHG18 do bioinformatic methods have to have all-caps names? Why are our method yelling?
— Testing enthusiast (@apicoplast) October 20, 2018

👢 🌊 👈🌊
🌊 🌊
🌊🌊🌊🌊 🌊🤠
🌊 🌊
👢🌊 👈🌊

HOWDY, I'M THE SHERIFF OF HORIZONTAL GENE FLOW #ASHG18
— The Sheriff of ASHG (@genome_sheriff) October 20, 2018

The end

And that’s a wrap!

LCG present at #ASHG18 🇲🇽 Great to catch up! @lcgunam @CinthyaZepeda @AleMedinaRivera @fellgernon @mariaGTAC @cgonzagaj pic.twitter.com/LadvPtaFur
— Claudia Gonzaga-Jauregui (@cgonzagaj) October 20, 2018

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

ASHG18 tweet summary day 4

Fri, 19 Oct 2018 00:00:00 +0000

Continuing from my ASHG18 day 1 post, day 2, day 3 here’s my list of tweets from day 4.

6F 11:00 am

Cecilia Lingdren

Got there at the end :P

Had #diversitymatters and many flags including the rainbow one in her last slide.

Benjamin Neale

.@bmneale points out dimorphism in research participation - women more likely to participate (generally and in UK Biobank) and yet comparatively understudied #ASHG18
— Avery Davis Bell, PhD (@averydavisbell) October 19, 2018

In the sex dimorphism session, @bmneale points out that in UKBB (as is true for many studies) there are more women than men. More women participate in research and remain in studies, but are actually studied less. #ASHG18
— Sara Pulit (@saralpulit) October 19, 2018

Benjamin Neale

UK Biobank has a great model for sharing data. Straightforward to request access and start collaborating with other human geneticists. @uk_biobank #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 19, 2018

Ben Neale @bmneale on Round 2 of @uk_biobank GWAS: Added new imputed data, chrX variants, and VEP missense and PTVs with MAF > 1e-6; net 3M more variants. Also ran sex-specific GWAS. Looking forward to updating Open Targets Genetics with this release! #ASHG18
— Ellen Schmidt (@ellenmschmidt) October 19, 2018

All results are available to download for free from his lab website!

Barbara Stranger

Barbara Stranger @be_stranger

Sex differences in: expression, splicing, interaction cOs-QTLs, allele-specific expression bias

Data from GTEx @GTExPortal

Work with @princyparsana @alexisjbattle among others #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 19, 2018

Other (morning)

Tissue cell type composition in Gtex changes with age, sometimes in a sex-specific manner #ASHG18
— Misha Vysotskiy (@Misha_Vysotskiy) October 19, 2018

Now: @julirsch does finemapping to find specific causal variants associated with blood cell traits. 230 genes contain fine-mapped coding variants #ASHG18
— Art Wuster (@artwuster) October 19, 2018

@princyparsana: "Addressing confounding artifacts in reconstruction of gene co-expression networks" https://t.co/9Tj85Furxd #ASHG18
— Damien C-C (@dccc_phd) October 19, 2018

M Snyder: Under GTEX consortium, the proteomics project mapped 32 normal human tissues, together have identified ~16k proteins from 14K genes. #ASHG18 #ASHG2018
— Dawei Lin (@iGenomics) October 19, 2018

Afternoon

I like Cecelia Lundgren’s emphasis on #mutualcontract of mentoring. Altshuler - “don’t bring the problem, bring the solution. Nice contrast to the typical “solve important problem” advice (which is also true).#ASHG18
— Sharon Plon (@splon) October 19, 2018

Cecilia Lundgren - it’s not “work-life” balance it is “work-home” balance as work is part of our lives! Love that!! She doesn’t email students over week-ends. She takes vacation. #ashg18
— Sharon Plon (@splon) October 19, 2018

CL: Science should embrace everyone, regardless of color, sexual orientation, religion, culture, etc. Diversity matters. A key fundamental part of genetics. [this was the best award recipient talk ever] #ASHG18
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 19, 2018

Really inspiring talk from Cecilia Lindgren accepting this year’s #ASHG18 Mentorship Award highlighting diversity, work-life balance, value of finding the best work environment for you. Science does not see sex, race, color #diversitymatters
— Claudia Gonzaga-Jauregui (@cgonzagaj) October 19, 2018

As a woman scientist it’s extremely inspiring to see women role models like this year’s #ASHG18 awardees @ceclindgren & Mary-Claire King and others like @splon @NancyGenetics share their stories, their wisdom, and their opinions #RepresentationMatters
— Claudia Gonzaga-Jauregui (@cgonzagaj) October 19, 2018

A moving acceptance speech by @ceclindgren on her #ASHG18 Mentorship award calling for inclusion, diversity, and assertiveness rather than aggression. Congrats! pic.twitter.com/3HQ5kle7Hu
— Chris Cole (@_chriscole_) October 19, 2018

CL: Advice:
- Lean in/hang on step up
- Be positive, persistent, polite
- Assertiveness ≠ aggression
- Choose a work environment supportive of you
- Choose a mentor supportive of you
- Choose a mentor that you can buy into and respect #ASHG18
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 19, 2018

Diversity

I prefer women/men or female/male - they’re not perfect since human gender and sex aren’t binary but they do a decent job as short hand. “Boys and girls” feels needlessly (verging in offensivey) infantilizing and perhaps extra gendered. #ASHG18
— Avery Davis Bell, PhD (@averydavisbell) October 19, 2018

Re-posting an example how I try to talk about (or avoid unnecessarily talking about) sex and gender in my own work. Not always feasible depending on topic and data, but always, always worth considering carefully. #ASHG18 (CC @fellgernon since related to previous discussion) https://t.co/edqepya3PH
— Avery Davis Bell, PhD (@averydavisbell) October 19, 2018

Misc

Still some people seams to have difficulties understanding no pictures means no pictures. Lady beside me has taken already 2 pictures of the slides and we are not even at the middle of the talk #ASHG18 https://t.co/8J7eggY0Sn
— Alejandra Medina-Rivera (@AleMedinaRivera) October 19, 2018

This got fixed super fast! Thanks for addressing the problem so quickly, @GeneticsSociety! #ASHG18 https://t.co/2sNJJypGje
— Jedidiah Carlson (@JedMSP) October 19, 2018

I can't attend the #ASHG18 Membership Forum. If you can, please propose that @GeneticsSociety appoint an awards nomination committee who will solicit or write nominations for worthy and overlooked members of our community. @nelsondl @NancyGenetics @splon
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 19, 2018

.@GeneticsSociety enjoy good locally roasted #coffee and cute #cats as part of your #ashg18 experience. We're a five minute walk from @sdconventionctr (cross Harbor at 1st, right on Island, walk to 3rd). Your admission includes a food or drink item. pic.twitter.com/tQZR0k8SB1
— Cat Cafe San Diego (@CatCafeSanDiego) October 19, 2018

Everyone should stop by @dnanexus booth to see the virtual reality system @marianattestad and her group has implemented. It’s stunning. They had to wrestle the headset away from me. #ASHG18
— Mark T. W. Ebbert (@bioinfo_mark) October 19, 2018

This is my personal account, if you've wondered who is doing all these tweets. Thanks again, Mary-Claire. #ASHG18 https://t.co/szITSzqKlK
— 23andMe Research & Therapeutics (@23andMeResearch) October 19, 2018

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

ASHG18 tweet summary day 3

Thu, 18 Oct 2018 00:00:00 +0000

Continuing from my ASHG18 day 1 post and day 2, here’s my list of tweets from day 3.

9:15 20BC

Jenna Carlson

Jenna Carlson: creating population-specific reference panels for improved genotype imputation #ASHG18
— Charleston Chiang (@CharlestonCWKC) October 18, 2018

.@jenccarlson Constructed a Samoan reference panel for imputation from 1,195 whole-genome sequenced Samoans as part of TOPMed. For MAF < 0.05, Ave R^2 0.696 with TOPMed ref panel, improves to 0.872 w Samoan-specific reference panel. #ASHG18
— Daniel E. Weeks (@StatGenDan) October 18, 2018

Christopher R. Gignoux

Next is Christopher Gignoux @popgenepi on the global landscape of pharmacogenomic variation as part of their work in the PAGE study #ASHG18
— 23andMe Research & Therapeutics (@23andMeResearch) October 18, 2018

Christopher Gignoux: revised version of Population Architecture using Genomics and Epidemiology (PAGE) study now on @biorxivpreprint https://t.co/t1x4qeGSkB #ASHG18
— Brooke LaFlamme (@Brooke_LaFlamme) October 18, 2018

CG: tested selection in pharmaco variants (haplotype length-based, corr with environment). Enviro-WAS takes geographic/climatic/ecological variables and cross with PAGE populations. #ASHG18
— Charleston Chiang (@CharlestonCWKC) October 18, 2018

www.pagestudy.org

CG (@popgenepi) describes our framework for conducting association analyses using phenotypes from NASA, World Wildlife Foundation, Berkeley Earth and GIDEON #ashg18
— Elena Sorokin, PhD (@ElenaSorokin) October 18, 2018

Angela Andaleon

“It’s my pleasure to introduce to you Dr Angela Andaleon” “Not a doctor yet!” “Aspiring doctor Angela Andaleon”

^_^

Angela is presenting the continuation of her undergrad project.

Angela Andaleon: transcriptome-based association study in hispanic cohorts implicates novel genes in lipid traits. #ASHG18
— Charleston Chiang (@CharlestonCWKC) October 18, 2018

AA: Replication cohorts: MESA HIS, GLGC. Ran gene-based association test in PrediXcan, using GTEx model and MESA model https://t.co/6wuD1fQB7l #ASHG18
— 23andMe Research & Therapeutics (@23andMeResearch) October 18, 2018

6C 10 am

Zongli Xu

Now: Z. Xu: Blood DNA methylation profiles are altered years before breast cancer diagnosis: Findings from a case-cohort analysis in the Sister Study #ASHG18
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 18, 2018

Lang Wu

Now: L. Wu: Genetically predicted methylation biomarkers and prostate cancer risk: A methylome-wide association study in over 140,000 European descendants #ASHG18 https://t.co/B4y1LUzB9m
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 18, 2018

LW: 1/Build CpG methylation prediction models with Framingham Heart Study data 2/Evaluate associations of genetically predicted methylation levels with prostate cancer risk using iCOGS-Oncoarray-GWAS meta results #ASHG18
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 18, 2018

Hmm, why use a 0.01 $R^2$ cutoff? Isn’t that super low? Maybe I missed something.

LW: 3/Significant associated CpG sites: assess correlations with expression of nearby genes in FHS. 4/Potential target genes: assess associations of genetically predicted mRNA expression with prostate cancer risk. #ASHG18
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 18, 2018

Moving to Hawaii to start his career as a PI. He had nice pictures to motivate others to join his lab ;)

11 am 6A

Hailiang Huang

Missed it: got there for the questions.

Tarjinder Singh

Tarjinder Singh

Looking at rare variants using exome sequencing that explain schizophrenia risk.
Genes with increased risk: TRIO (loss of function) and (missed it), GRIN2A (example of common and rare variant info being concordant).

#ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 18, 2018

They built a browser for seeing the rare variants and the exome data. Not available yet.

Rujia Dai

Rujia Dai

Cell types + schizophrenia disorder + bipolar disorder. Used psychENCODE data

Estimated cell types in different brain regions. Identified cell type-specific co-expression modules.

#ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 18, 2018

Rujia Dai

Oligodendrocyte, microglia, neuron upper cortex, neuron deeper cortex, astrocytes

SCZD and BP are more related in the neurons from the upper cortex #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 18, 2018

Ying Ji

Used GTEx and BrainSpan data.

Step 1. Identified risk genes using a method from an upcoming paper at Nature Genetics by Quan Wang, Rui Chen, et al.

Step 2. Genome wide prediction: risk and background genes. #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 18, 2018

Ying Li

Gpred and Bpred genes (GTEx and BrainSpan) enriched for SCZD gene lists. Small overlap between them.

Used CommonMind Consortium SCZD DEGs list. Bpred tend to be upregulated in SCzD cases.#ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 18, 2018

Bpred and Gpred genes have different expression over development: they intersect at birth.

Kiran Girdhar

Kiran Girdhar

H3K27ac and H3K4me3 modifications (ChIP-seq) + flow sorted neurons, non-neurons and homogenate data

See Girdhar et al, Nature Neuroscience (I think), 2018

#ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 18, 2018

Kiran Girdhar

Used standard peak calling software.

Explored variation explained for each mark.

Performed differential binding analysis for each mark among SCZD cases and controls. #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 18, 2018

Looked at enhancers too.
Question from the audience about the differential shapes of the peaks between cases and controls.
Will look at QTLs after ASHG18.

Siwei Zhang

Looked at SNPs associated with ATAC-seq data (ASoC ?).

Used CRISPR to perturbe noncoding variants linked to SCZD risk: the ASoC SNPs.

Then used single cell (10x Genomics) to measure expression.

Siwei Zhang

Looked at SNPs associated with ATAC-seq data (ASoC ?).

Used CRISPR to perturbe noncoding variants linked to SCZD risk: the ASoC SNPs.

Then used single cell (10x Genomics) to measure expression. #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 18, 2018

Other (morning)

Andy Clark using microbiome to improve GWAS analysis and "this is getting a bit geeky for those that like linear mixed models"... #ASHG18
— Ana Viñuela (@anavinuela) October 18, 2018

E. Davenport: phenotype ~ covariates + relatedness + microbiome <= model microbiome beta diversity as an additional random effect. Biggest microbiome var up to 25% for height, weight, BMI. Controlling for microbiome can improve power to detect phenotype/SNP association. #ASHG18
— Daniel E. Weeks (@StatGenDan) October 18, 2018

Afternoon awards

Eric Lander

EL: The challenge ahead is to move rapidly from variants to function. "I don't want to just tell history - I think we are entering into a period that is as generative...as the period where much of the foundation was laid" #ASHG18
— 23andMe Research & Therapeutics (@23andMeResearch) October 18, 2018

@eric_lander on career planning to trainees like myself: don’t worry too much it will mostly be happy accidents.
already as a first year grad student thankful for a list of people that have led me here. #ASHG18
— Rachel Ungar (@raungar) October 19, 2018

Jan M. Friedman

Jan Friedman is the recipient of the Arno Motulsky-Barton Childs Award for Excellence in Human Genetics for increasing genetic education in medical schools #ASHG18
— 23andMe Research & Therapeutics (@23andMeResearch) October 18, 2018

JF: When the society was first founded, education was not a major focus. Focus was on research and on the journal publishing it. Appropriate to focus on this after WWII after the "perversion of genetics" that lead to eugenics laws - had to work to repair genetics #ASHG18
— 23andMe Research & Therapeutics (@23andMeResearch) October 19, 2018

JF: HJ Mueller: "The two major purposes of [ASHG] were to be the furtherance of sound research in [human genetics] and the publication of [AJHG]." No mention of education. #ASHG18 https://t.co/TAp5LZzgdu
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 19, 2018

Jan Friedman tells us that the medical board exams had no medical genetics questions in the 1990s. The only mentions of genetics were distractors (wrong answers). He got together a group to fix it. #ASHG18
— Avery Davis Bell, PhD (@averydavisbell) October 19, 2018

Presidential Symposium

John Hawks

Starting now the #ASHG18 Presidential address with @johnhawks speaking first about human origins
— Liz Crocker (@liminalphase) October 19, 2018

John Hawks: Fossils are rare, and it is hard to say novel things about them if you don't find them. So I did what any of you would have done; went into genetics! #ASHG18
— Joanna Patterson-Cross (@jpcphotouk) October 19, 2018

.@johnhawks Three main points 1) humans emerged over a long period between 200-350kya 2) there was deep long lasting diversity across that timespan 3) pops w/ very ancient roots survived until very recently & we don't fully know their relationship(s) to us #ASHG18
— Liz Crocker (@liminalphase) October 19, 2018

Human evolution definitely absolutely most certainly doesn’t look like those t-shirt cartoons. @johnhawks just introduced us to MANY diverse hominins. So cool to meet you, guys. #ASHG18
— Dr Julija 💉💉💉💉 Hmeljak (@JHmeljak) October 19, 2018

Himla Soodyall

Now: Himla Soodyall: Reconstructing the prehistory of Africa: the narrative of the genes. "Greetings, fellow Africans." #ASHG18
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 19, 2018

Himla Soodyal: Theory of “Out Of Africa” origins for the human species dismisses that evolution in Africa is forever continuing. #ASHG18 “It’s time to celebrate Africa”
— Claudia Gonzaga-Jauregui (@cgonzagaj) October 19, 2018

HS: Genomic Variation in Seven Khoe-San Groups Reveals Adaptation and Complex African History. STRUCTURE plots. There are no "pure" populations. "We are all mongrels." cc @amy_harmon #ASHG18 https://t.co/Psu3b2lRe3 pic.twitter.com/ILvGcksS4I
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 19, 2018

Appreciate reminder to think about the societal value of science, and to conduct yourself (and the research) accordingly. #ASHG18
— Cathy Long (@yhtacgnol) October 19, 2018

Ambroise Wonkman

AW: Story of the curse of Chief K. Actually turned out to be Fragile X Syndrome. #ASHG18
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 19, 2018

AW: researchers coming to Africa for genetics research are mostly interested in population genetics, not genetic medicine #ASHG18
— Genome Biology (@GenomeBiology) October 19, 2018

Diversity

The diversity of participants we see at #ASHG18, shows how different we are, yet similar and genetically the same as #Human #HumanGenetics #UnityinDiversity
— Dr. Swarkar Sharma (@sawerkar) October 18, 2018

.@simon_r_myers standing in for Sinan Shi in session 43 because speaker couldn’t get his visa. Hearing this far too much 😞 #ASHG18
— Brooke LaFlamme (@Brooke_LaFlamme) October 18, 2018

#MAGA deal with it, foreigner.
— Tommy (@tomsegal) October 18, 2018

Trolls :/

This is sort of a silly thing to say, but after attending so many conferences that are predominantly male (cough, JP Morgan, cough) it’s actually nice to wait in line, behind so many scientists, in the ladies room. #ashg18
— Meghana Keshavan (@megkesh) October 18, 2018

Echoing yesterday's NSGC GC roundtable, a reminder that genomics is failing diversity. How do we do better? #ASHG18 #gcchat
— Cathy Long (@yhtacgnol) October 19, 2018

Misc

May recommend speakers not perform an awards ceremony for “everyone that made this research possible”? Not a great use of time.

Put their names up, bold them, or even include pictures.

I liked how @dgmacarthur lab included pictures of individuals on specific slides.#ASHG18
— Mark T. W. Ebbert (@bioinfo_mark) October 18, 2018

Good morning #ASHG18 tweeps! Help us spread the word about our travel grants for #ECRs. Award will be paid soon after winner announcement. No need to wait for reimbursement. Applications due Nov 5 https://t.co/JoJxIGwY9b
— Brooke LaFlamme (@Brooke_LaFlamme) October 18, 2018

This is hard. Some don’t mention the methods to avoid this issue, but then it becomes a “black box”. I guess that it can also be a “black box” if they don’t explain a bit what the model is/does. /shrug

*takes a big ol' bite of beef jerky* REMEMBER FOLKS, TAKIN' PHOTOGRAPHS O' SOMEONE ELSE'S SLIDES WITHOUT PERMISSION IS A BREACH OF PROFESSIONAL ETHICS #ASHG18 https://t.co/CG0Y1pCvQ1
— The Sheriff of ASHG (@genome_sheriff) October 18, 2018

#ashg18 #ASHGtrainee For presenters, this might be helpful especially for the big rooms:https://t.co/KVuIXRytYW
— Alon Goren 🇺🇦 (@alon_goren) October 18, 2018

Maybe too soon to say this, but @genome_sheriff is the best emoji-based wild west hyperbole novelty account I've ever seen at a genetics conference #ashg18
— Erick Loomis (@ErickLoomis) October 18, 2018

"our code is on github and our preprint is up on bioRxiv" - things I love hearing during #ASHG18 talks. Instant credibility boost.
— Daniel MacArthur (@dgmacarthur) October 18, 2018

On our way to the presidential address. #scooters #ASHG18 pic.twitter.com/UrtAuoD2jd
— Audrey Hendricks (@hendricks_ae) October 19, 2018

Still can't get over that almost 9,000 people are here! Absolutely amazing. #ASHG18
— John Morris (@johnomix) October 19, 2018

As always please feel free to correct anything I got wrong! Thanks @splon #ASHG18 https://t.co/hmRGx5CuM1
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 19, 2018

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

and everyone that I met at the tweetup

#ASHG18 pic.twitter.com/8RJh35VzeO
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 19, 2018

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

ASHG18 tweet summary day 2

Wed, 17 Oct 2018 00:00:00 +0000

Continuing from my ASHG18 day 1 post, here’s my list of tweets from day 2. Note that I changed sessions a few times.

I have to say, digitally attending #ASHG18 by frantically refreshing the hashtag feed in my living room is...not quite the same. But maybe I'll get fish tacos for dinner.
— Julie Nadel (@JulieNadel) October 17, 2018

Turn down the lights and turn up the AC for the fuller remote #ASHG18 experience! 🤪
— Damien C-C (@dccc_phd) October 17, 2018

6E 10:30 am

Live tweet our #ASHG18 session on #SilentGenomes to hear what #Indigenous scientists are doing to change the narrative and push for more #IndiGenomics moderated by @KeoluFox and @NanibaaGarrison on Wed at 10:30am PT https://t.co/zI8NPXotop
— Nanibaa' Garrison (@NanibaaGarrison) October 16, 2018

Nanibaa’ Garrison

Check Claw et al paper at @NatureComms from @SINGConsortium

“A framework for enhancing ethical genomic research with Indigenous communities.” https://t.co/qBuDWGPIV4 #indigenomes #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 17, 2018

Maile Taualii

“Nobody wakes up thinking how can I harm indigenous populations!”

Maile Taualii: no one goes into research with the intent of harming indigenous communities, so why does this happen? Publish or Perish causes people to rush, skip steps. Ignorance of issues (not an excuse). “For their own good” mentality #ASHG18 #SilentGenomes #IndiGenomics
— Brooke LaFlamme (@Brooke_LaFlamme) October 17, 2018

Don't get distracted by the sexiness of genetic science! (Maile Taualii) #silentgenomes #ASHG18 #indigenomics
— Nanibaa' Garrison (@NanibaaGarrison) October 17, 2018

#ASHG18 #silentgenomes Maile directly addressing the "residue" of colonialism that is linked to culturally sustainable biomedical research today!
— Dr./Prof. Keolu Fox (@KeoluFox) October 17, 2018

Taualii: take-home messages
1. There is no easy solution
2. Ignorance isn’t an excuse
3. Understand what is at risk
Think about whether your work is going to benefit the generations to come #SilentGenomes #IndiGenomics #ASHG18
— Brooke LaFlamme (@Brooke_LaFlamme) October 17, 2018

Nadine Caron

Nadine Caron: addressing inequity in genomic diagnosis & research. #ASHG18
— Charleston Chiang (@CharlestonCWKC) October 17, 2018

Nadine Caron: "It is known that those who have the greatest health disparities, will actually benefit the least from new technologies and research" #ashg18 #silentgenomes
— John Hawks (@johnhawks) October 17, 2018

I think that Nadine Caron’s suggestions for First Nations and indigenous populations in Canada apply to other countries and underrepresented communities #IndiGenomics #silentgenome #ASHG18

Right @paleogenomics? https://t.co/7m5eWNpC8r
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 17, 2018

Agree!! That is an approach to follow with vulnerable communities. e.g. #Afromexicans, they are not only underrepresented but also have been sadly neglected for centuries.
— paleogenomics (@paleogenomics) October 17, 2018

Nadine Caron - access to research is an important determinant of health. Such a critical issue for health equity issues across the world. #silentgenomes #ASHG18
— Chris Gignoux (@popgenepi) October 17, 2018

Ngiare Brown

N Brown: Ancestry is not the same as culture. Genetic information today is increasingly commercialized and is creating another layer in an already tangled web #silentgenomes at #ASHG18
— Elena Sorokin, PhD (@ElenaSorokin) October 17, 2018

Dr. Ngiare Brown: “Some information cannot be individually owned.” Genomics research w/indigenous communities needs to involve to thinking beyond Western notions of consent. #SilentGenomes #ASHG18 #IndiGenomics
— Riley Taitingfong, PhD (@riley_ilyse) October 17, 2018

We too (Australia Aborigines) are excited about the sexiness of genetic research. So how do we make sure it gets done right? (Ngiare Brown) #SilentGenomes #ASHG18
— Nanibaa' Garrison (@NanibaaGarrison) October 17, 2018

Third time she mentioned “sexy research”. I thought that this term was frowned upon, maybe it’s not.

Maui Hudson

He made us laugh with his conflict of interest slide. No commercial interests. Just interest in indigenous genomics and helping others.

I like how Hudson is framing the scientific contribution of New Zealand: The genomic resources of Māori people are a unique resource, partnering with the community to enable them to be part of the conversation is essential to good science. #ashg18 #silentgenomes
— John Hawks (@johnhawks) October 17, 2018

Issue with open data. They want to control how the data is re-used. Like for imputation: what are they looking at? Is it something they agree?

Hudson is expressing, though not in so many words, a tension between the consultative approach, in which indigenous communities have a voice in how data are used, and what human geneticists have conceived as "open access" to genome data. #ashg18 #silentgenomes
— John Hawks (@johnhawks) October 17, 2018

I am struck by questions from audience members at #silentgenomes panel. Geneticists who are already working on projects with indigenous subjects, and they are ONLY NOW asking how they can seek involvement from those communities. #ashg18
— John Hawks (@johnhawks) October 17, 2018

These are well-intentioned, nice geneticists, I can see. But if you're asking a panel of indigenous scientists from other parts of the world to speak for the communities that you are working with, you're doing it wrong. Still, this is why #silentgenomes panel matters. #ashg18
— John Hawks (@johnhawks) October 17, 2018

6C 11:30 am

Kelly East

Kelly East of @hudsonalpha: data from Concert Genetics suggest on average, 10 new genetic testing products enter the market every day. #ASHG18
— Chris Gunter (@girlscientist) October 17, 2018

KE discussing how medical genetic testing is rapidly expanding both in types of tests and patients being tested. We don’t have enough genetic providers to handle this, and the ones we do have aren’t uniformly distributed. #ASHG18
— Avery Davis Bell, PhD (@averydavisbell) October 17, 2018

Kelly East wants to train MD, NP, PA so that they can understand genetics more and help patients too #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 17, 2018

(others made better tweets!)

K East: how do we bring more genetics services to everyone? 1. Make more providers. 2. Increase capacity of existing genetics providers. Or 3, her favorite: Train other healthcare providers to do genetics. #ASHG18
— Chris Gunter (@girlscientist) October 17, 2018

Solution to this KE is advocating is training non-genetic healthcare providers to do the genetics to help patients who will never make it to a genetic counselor. Many providers already interested in genetics. #ASHG18
— Avery Davis Bell, PhD (@averydavisbell) October 17, 2018

Given that 52% of #GeneticCounselors are in 10 states, & 60% are in metro areas: We need more genetic counselors, we need GC assistants, we need streamlining and improved efficiency AND let's talk about more genetic training for MDs, NPs, and PAs! @Kelly_M_East #ASHG18
— Sally G Pasion (@sgpombe) October 17, 2018

KE how to educate these providers? Case based, inter professional, and electronic/web based - good opportunitieshere especially as these platforms already heavily used in ongoing medical training #ASHG18
— Avery Davis Bell, PhD (@averydavisbell) October 17, 2018

Kelly East: a lot of opportunity for developing web based activities to help train non-genetics colleagues (who are hard to get into a room given the nature of their work/schedules) #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 17, 2018

.@Kelly_M_East developed a toolkit that is hosted on the ASHG website https://t.co/LvcCzpHjn9 for training non-genetics providers.

Feedback: good genetic info, but need more info on suggested next steps (which KE didn’t want to have initially in respect of colleagues) #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 17, 2018

Hmm, can’t say that I’m a fan of the “SouthSeq” project name. Though I don’t have an alternative name on the top of my mind @Kelly_M_East #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 17, 2018

I guess that it’s for US scientists, but I don’t think of the “Deep South” in the US when I hear “SouthSeq”. Anyway, SouthSeq looks like a great project with many ways to train others in genetics.

“We are not going to run genetic providers out of jobs”. Loved this quote!

Chris Gunter

Chris Gunter @girlscientist will close out the discussion with a call to action - doing science is not enough, you have to communicate your research to the public in any way you can #ASHG18
— Avery Davis Bell, PhD (@averydavisbell) October 17, 2018

35,000 page views on DNA awareness day3. 23andMe did a survey study in March 2018

Chris Gunter: your paper is a nucleus for a larger #scicomm ecosystem #ASHG18
— Kathryn Vaillancourt is writing all the things (@MoleculeMind) October 17, 2018

Science literacy does NOT equal support for science. New framework: Science ENGAGEMENT -- a two-way street. @girlscientist #ASHG18
— Cincy Kids Genomics (@CincyKidsGenomX) October 17, 2018

@girlscientist We MUST think about the sociopolitical climate we are sharing knowledge in. This stuff isn’t neutral. #ASHG18
— Avery Davis Bell, PhD (@averydavisbell) October 17, 2018

Ohh! #medRxiv will be online soon! Looks like it’s related to @biorxivpreprint (from the website design) right @cshperspectives? @girlscientist announced it at #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 17, 2018

Follow @medrxivpreprint!! #ASHG18 https://t.co/sanAbTK1lI
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 17, 2018

@girlscientist just gave an interesting talk about providing scientific education to general public. Don't think it's as simple as a knowledge deficit issue - much more nuanced and complex. (Just like humans?) #ASHG18
— Cathy Long (@yhtacgnol) October 17, 2018

Hmm, I get the point that a republican scientist might have an easier time communicating with a republican audience. But it also felt discouraging to me. Well, I guess that you should always try to be aware of your audience and try your best to connect with them. #ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 17, 2018

Beth Tuck: @genome_gov would love to have everyone as partners for DNA Day 2019. Check out their site at https://t.co/DQUWzVO5bB, of their Pinterest, or Twitter, or Facebook, or suggest something new! #ASHG18
— Chris Gunter (@girlscientist) October 17, 2018

Other (morning)

BM: Assumptions $45,000 max for sharing by publisher, recipient pays $180 for data, receives reward of $360 for successful detection. Allow 10% FPR, 50% TPR. Prior of 0.05. #ASHG18 https://t.co/1KYmCaexDb
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 17, 2018

"For non-scientists like lawyer, cryptographic keys are like dark magic. Explain properly those technology is part of the #privacy challenge." @erlichya #ASHG18 #ASHG2018
— Guillaume Dumas (@introspection) October 17, 2018

20BC 4:15 pm (4:23)

Semanti Mukherjee

In session #25: Integrated Variant Analysis in Cancer Genomics. #ASHG18 https://t.co/O3p4JElpB5
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 17, 2018

Semanti Mukherjee: 8.5% of patients in an unselected lung cancer cohort had germline mutations in known cancer susceptibility genes including BRCA1/2, CHEK2, P53, MUTYH, and others including highly penetrant variants and low penetrance susceptibility variants #ASHG18
— Claudia Gonzaga-Jauregui (@cgonzagaj) October 17, 2018

Chimene Kesserwan:

Now: Chimene Kesserwan: Assessing causality of pathogenic and likely pathogenic germline variants by integrating somatic and germline sequencing in children with cancer enrolled on the “Genomes for Kids” (G4K) sequencing study at St. Jude Children’s Research Hospital #ASHG18
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 17, 2018

Cynthia Zepeda

Next @CinthyaZepeda presenting Beyond FISH and Karyotype detecting complex rearrangements by mate-pair sequencing in leukemia and lymphoma #ASHG18 @lcgunam
— Claudia Gonzaga-Jauregui (@cgonzagaj) October 17, 2018

CZ: Going beyond karyotype and FISH using MPseq that identifies chromosomal abnormalities at high resolution in leukemia samples. Identified ~110 abnormalitiesper case in a cohort of 16 B-ALL cases MPseq helped refined karyotype calls #ASHG18
— Claudia Gonzaga-Jauregui (@cgonzagaj) October 17, 2018

Xiaowu Gai

He thanked others for uploading data and making it publicly available so bioinformaticians like him can download it and re-analyze it. I totally agree with him!

6C 5:15 pm

Julio Barrera

Replaced by Chiba-Falek (lab PI).

Single cell (Neuron/glia)( ATAC-seq + RNA-seq + case-control data studying LOAD. RNA + ATAC matches.

Yungil Kim

PI Panos Roussos. Single cell data.

Other (afternoon)

Kudos @SarahTishkoff for highlighting the -most- informative single measure of a single human genome: # of heterozygous sites. #ashg18
— Nathan Pearson (@GenomeNathan) October 18, 2018

20 BC 6 pm

Christopher Lau

Chris Lau from @genome_gov now presenting on genes of unknown significance in 20BC to kickoff session 34- reanalysis of sequencing data to increase dx yield #ASHG18
— UDN (@UDNconnect) October 18, 2018

CL: Discussing use of the UDN to match patients with variants impacting the same amino acid in a gene that was not a known disease gene at the time; simply prioritised due to nature of the impact. Chao and Davis et al AJHG 100 2017. #ASHG18
— Liz Worthey (@lizworthey) October 18, 2018

Matt Velinder

Up now Matt Velinder discussing iterative reanalysis provides diagnostic avenue for previously unsolved rare and complex disease cases #ASHG18
— Liz Worthey (@lizworthey) October 18, 2018

MV: 44 patients have been accepted into their penelope rare and undiagnosed disease program (out of 70 who applied). Trio WES gave 17 diagnoses, 6 probable diagnoses, and 16 no diagnoses #ASHG18
— Liz Worthey (@lizworthey) October 18, 2018

Matt Velinder presented results processed using: (they are URLs)

vcf.iobio bam.iobio echo.iobio

Part of the Undiagnosed Network (UDN) @UDNconnect

#ASHG18
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 18, 2018

20 A 6:42 pm

Ira Hall

Ira Hall: As SV population frequency goes down size goes up. Ultra rare SVs are larger than common SVs. #ASHG18
— ryanlayer (@ryanlayer) October 18, 2018

Controlling for a nearby pLI gene? I’ve seen pLI mentioned a few times. Gotta read more about it!

Sarcasm/fun

Massively parallel biology students.... #ASHG18
— Tricia Hall (@BiochemG_thall) October 17, 2018

There should be a time penalty for #ASHG18 speakers who use fonts below a certain size. #usebiggerfonts #morepicturesfewerwords
— Chloe Reuter (@ChloeReuterCGC) October 17, 2018

#ASHG18 Best exhibitor presentation I saw was “Long reads? You don’t need no stinking long reads.” at the Illumina booth.
— Testing enthusiast (@apicoplast) October 17, 2018

TFW you're in an overflow room, and the speaker says 'thank you for listening, I'll take questions', and you automatically, absentmindedly start clapping, and nobody else in the room does #ASHG18
— Genome Biology (@GenomeBiology) October 18, 2018

Needed a last min bday present for my brother’s bday dinner tonight. Good thing I’m at #ASHG18 where @23andMe is selling these kits. #iforgetbirrhdays #badsibling #geekypresents pic.twitter.com/eifRa4gCMK
— Dr. Joyce Kao (@joyceykao) October 18, 2018

plus they have a 50% off discount

Beginning to think I chose my session attendance poorly. I might be in a broom closet. #ASHG18
— Testing enthusiast (@apicoplast) October 18, 2018

⠀ ⠀ ⠀ 🤠
　 💯💯💯
💯 💯　💯
👇 💯💯 👇
　 💯 💯
　 💯　 💯
　 👢 👢

⠀ ⠀ ⠀ 🤠
　 💯💯💯
💯 💯　💯
👇 💯💯 👇
　 💯 💯
　 💯　 💯
　 👢 👢
HOWDY, I'M THE SHERIFF OF REPRODUCIBLE RESEARCH #ASHG18
— The Sheriff of ASHG (@genome_sheriff) October 18, 2018

Diversity

@SarahTishkoff now presenting on behalf of a former postdoc who couldn’t get a visa (!!! This is such a problem and not unique) #ASHG18
— Avery Davis Bell, PhD (@averydavisbell) October 18, 2018

So lucky to hear @SarahTishkoff speak at #ASHG18 but heartbroken she’s subbing in bc of visa issues. Isolationism and strict immigration policies are not how we keep pursuing scientific progress.
— Natalie Telis (@NatalieTelis) October 18, 2018

Just putting this out there for #ASHG18 :D https://t.co/LuG14mYpbD
— Liz Worthey (@lizworthey) October 18, 2018

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

and everyone who wrote tweets with the ASHG18 hashtag!

ASHG-attending tweeps: hope to meet you at the #ASHG18 Tweetup! 7:30p Thursday, Garage Kitchen+Bar, 655 Fourth Ave. https://t.co/kfYRQSlTKt
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 14, 2018

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

ASHG18 tweet summary day 1

Tue, 16 Oct 2018 00:00:00 +0000

Today was the first day of the American Society of Human Genetics (ASHG) 2018 conference. The official hashtag for the conference is ASHG18 on Twitter. At first I was tweeting myself and checking both the top and the latest tweets. As the day progressed I started a Google Doc to take notes during talks. I was missing some details so I was relying on the latest tweets and copy-pasting the tweet links to my notes. At some point I told myself I should simply turn this collection of tweet links into a simple blog post. So here it is for day 1. You can consider it a curated list of the ASHG18 tweets. Although it’s incomplete because it only covers the talks I went to starting from the Presidential address.

The linked tweets are frequently from Eli Robertson¹ and Michael Hoffmann², but also from other Twitter users³.

So, without further ado, here we go:

Arrive in SD for @GeneticsSociety #ASHG18

Cab driver a Greek Orthodox Palestinian from Jerusalem
Tell him that I am here for genetics meeting

And his 1st ?: "Is the ancestry stuff accurate?...@Ancestry test told me that I'm 27% Greek and I know I am more than that."

😳
— Sek Kathiresan MD (@skathire) October 16, 2018

Award presentations

JL: Population → clan → family → patient. #ASHG18
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 17, 2018

Sek also thanked all his mentees ^^.

#ASHG18: @gabecasis presenting the Curt Stern award to @skathire: “...be like Sek: think big, be enthusiastic, and self-examine”.
— Maria Chahrour (@MariaChahrour) October 17, 2018

Diversity

Thanks to @GeneticsSociety for expanding the code of conduct for #ASHG18. Let's make the meeting safe, productive, and fun! https://t.co/CLu84vonSD pic.twitter.com/lT4bycqHxG
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 14, 2018

On the way to #ASHG18 to learn, meet, and share! interested in knowing how latent artifacts affect reconstruction of coexpression networks? Come to my talk on Fri, Room 6C#womeninSTEM #scientistMom
cc Claire Ruberman, .@jtleek @alexisjbattle @andrewejaffe @mike_schatz pic.twitter.com/n3VrclqHS3
— Princy Parsana (@princyparsana) October 16, 2018

3.75% of attendees are Hispanic at #ASHG18 (hosted in San Diego this year)

Would love to see more! I know @lcgunam is training a few dozen a year and @sacnas is helping thousands across science.

Let’s keep improving!#diversity
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 16, 2018

Mine is simple. I don’t know everything @GeneticsSociety does to promote #diversity, but it’d be great if they took a 👀 at @rstudio’s diversity scholarship. Membership fee can be too prohibitive for scientists outside the US (if they manage to cover ✈️) https://t.co/nmZY7UrQUf
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 16, 2018

Sek @skathire starts his #ASHG18 Curt Stern award presentation by briefly mentioning his immigration to the US story and highlighting it all in one word:

“possibility”

👏🏽🙌🏽
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) October 17, 2018

Mitchel Cole

SNP crispr-cas9 perturbation study in blood cells.

#ashg18 Mitchel Cole describing genomewide CRISPR KO screen using red blood cell trait GWAS hits from our 2016 Cell paper using @uk_biobank & INTERVAL data: https://t.co/Ir4gk2rZ88
— Adam Butterworth (@aidanbutty) October 17, 2018

MC mentioned metaFDR for combining pvalues from 4 tests.

Priyanka Nakka

Uniparental disomy using 23andMe data. Seeking prevalence information in general population.

PN: Open questions: 1/What is the prevalence of UPD in general population (not ascertained for disease)? 2/What are rates of maternal UPD, paternal UPD, and subtypes in general population? 3/What phenotypes are associated with UPD? #ASHG18
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 17, 2018

3 types that can be identified by 2 in silico methods.

PN: Used @23andMeResearch data, 5M+ customers, 80% consented for research. Computationally detect 3 subtypes of UPD (heterodisomy, isodisomy, or partial isodisomy) based on IBD between parent-child and ROH analysis. #ASHG18
— Charleston Chiang (@CharlestonCWKC) October 17, 2018

PN: In ~900k duos in the 23andMe database, find 199 individuals with UPD. Overall rate of 1/2000 - twice as common as previously thought & first estimate of UPD prevalence in general population #ASHG18
— 23andMe Research & Therapeutics (@23andMeResearch) October 17, 2018

PN: Problem distinguishing between UPD and consanguinity in parents. Trained logistic regression classifiers for each chromosome and population separately #ASHG18
— 23andMe Research & Therapeutics (@23andMeResearch) October 17, 2018

Mentioned PheWAS.

PN: Don’t know if you can get single cell data from 23andMe customers.

If you're interested in interning with us like Priyanka did, send me a DM! You can also check out the current job openings on our Research Team here: https://t.co/TlJVQjdjqT #ASHG18
— 23andMe Research & Therapeutics (@23andMeResearch) October 17, 2018

If like the previous question-asked you’re interested in single cell sequencing of sperm, you MUST check out @averydavisbell’s talk this Thursday at 9 am! #ASHG18
— Beryl Cummings (@beryl_bbc) October 17, 2018

I was too busy being a #proudPI to live tweet Priyanka Nakka’s talk about work with @23andMeResearch on identifying cases of uniparental disomy using IBD and runs of homozygosity. Find Priya during #ASHG18 to find out more!
— Sohini Ramachandran (@s_ramach) October 17, 2018

Jack A Kosmicki

Next up, my lab big brother @Jack_Kosmicki presenting on the latest in autism gene discovery on behalf of the Autism Sequencing Consortium #ASHG18
— Beryl Cummings (@beryl_bbc) October 17, 2018

mentioned pLl method

“@ksamocha does everything”. #ASHG18
— Eugene Gardner (@DrGeneUK) October 17, 2018

.@Jack_Kosmicki #ASHG18: Constraint scores (MPC for missense and pLI for loss-of-function) help identify de novo variants enriched in cases with autism.

Also thanks for the shout-outs, Jack. :)
— Kaitlin Samocha (@ksamocha) October 17, 2018

99 ASD genes

.@Jack_Kosmicki : 99 genes at FDR of 10%, 25 of which are genome-wide significant #ASHG18
— Beryl Cummings (@beryl_bbc) October 17, 2018

JK: 41/50 ASD-preferential genes have only 0-1 de novo missense or premature termination variants in ID/DD #ASHG18
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 17, 2018

summary

JK: some de novo variants for autism are even more enriched in non-autistic intellectual/developmental delay. Others are ASD preferential. Stronger neg selection in ID/DD- than ASD-preferential genes. ID/DD-preferential genes associated with walking later. #ASHG18
— Daniel E. Weeks (@StatGenDan) October 17, 2018

Caitlin Bowen

Sorry, I left this talk early to see the poster talks (that I didn’t know had been delayed).

Wrap up

Poster talks

#ASHG18 “Poster Talks” starting in a few mins!! Get a sneak peek at some of the best data being presented this year @GeneticsSociety via these posters nominated to give brief talks. Here’s one from @Genomes2People @RobertCGreen: https://t.co/dEI5iOXCf9 Ballroom 6C pic.twitter.com/GIELxjyRps
— Genomes2People (@Genomes2People) October 17, 2018

Single cell composition and organs poster 2012 seemed interesting to me. I added to my calendar!

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
knitcitations (Boettiger, 2021)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)

as well as everyone who was tweeting! See you on Thursday!

ASHG-attending tweeps: hope to meet you at the #ASHG18 Tweetup! 7:30p Thursday, Garage Kitchen+Bar, 655 Fourth Ave. https://t.co/kfYRQSlTKt
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) October 14, 2018

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Eli Robertson creates great Twitter threads, so I frequently only link to the first tweet of the thread. ↩︎
Michael Hoffmann also writes multiple tweets per talk, though you’ll have to scroll through his timeline to find all the ones related to a talk. Unless I’m missing a way to make them into a thread. ↩︎
Some with a few followers, some with many. It didn’t matter. I was just checking the latest and saving the ones I liked the most. ↩︎

Why I applied to join SNI: the Mexican National Researchers registry

Sat, 29 Sep 2018 00:00:00 +0000

A few days ago a friend of mine told me that I was on the list of newly admitted SNI members. A few have asked me since why did I request to join it. So here’s my public reply.

Woo! Ya soy "Investigador Nacional Nivel I" en el Sistema Nacional de Investigadores de CONACyT en México @Conacyt_MX 🎉💪🏾🇲🇽

I'm a National Researcher lvl I in the Mexican National Researchers Registry ^_^ 🎆🎉https://t.co/hud7Z22WLY

Gracias @malacopa_genome por el aviso! pic.twitter.com/NWnf90KOF1
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) September 26, 2018

SNI

First let me tell you what SNI is. It stands for Sistema Nacional de Investigadores which translates to National Researchers Registry. It’s a way that the Mexican government created in 1984 to recognize high quality researchers and encourage them to continue to do great research, teach the next generation of researchers and promote innovation (Wikipedia). There are three types of members:

Candidato a Investigador Nacional: Candidate to National Researcher
Investigador Nacional: National Researchers with 3 levels (1, 2 and 3)
Investigador Nacional Emérito: National Emeritus Researcher

To become a member you have to submit documentation to CONACyT showing that you have made contributions to science, technology, culture and society through scientific or technological developments, training new students, popularizing or translating science and technology, among other criteria. Once you are a member, you have to keep submitting new documents every couple of years to stay a member or advance in the ranks.

The real reason behind joining SNI

I’ve read a little bit that there are many critics of the SNI system. Most of the criticisms are based on how they decide what is a good contribution and how they judge applications. The reality is also that the process of sending the necessary documentation is cumbersome to say the least. The CONACyT website breaks often enough that it can be painful to use.

So why do people want to join the SNI and go through all this complicated process? The main reason (I think) why people want to join the SNI is because being a member comes with a financial supplement. It’s enough that I’ve heard that it can constitute about a third of your salary. You can find headlines such as how CONACyT’s support to researchers went from a priviledge to a salary. This financial supplement is given to those members that work 20 hours a week on Mexican institutions.

Given how important this boost in salary is, I can understand very well the frustrations that come with interacting with the CONACyT website: specially when the website goes down and you have to submit your application before a given deadline.

So why did I apply to join SNI?

You can find on CONACyT’s website an interview on why it’s important to join SNI or go elsewhere to read a study on the motivations people have for joining SNI. But why did I apply to join SNI given that I’m outside Mexico and won’t be getting the financial supplement?

My main motivation was protection.

Lets say that I return to Mexico as a researcher at UNAM or another Mexican institution. Currently there is only 1 call for new SNI members per year. If I were to return today (September 2018), I would need to wait several months for the 2019 application cycle. Then I would need to wait several months for all applications to be evaluated and would probably start receiving the financial salary boost on January 2020. That is a lot of time to wait!

Furthermore, right now I don’t need the salary boost. But if I were to return then it could be a significant portion of my salary. Having all this riding on a cumbersome application (I’d be afraid of making a silly mistake in the application) and on an external evaluation would be stressful.

So the stakes for me were low right now. Now that I’m a SNI member if I were to return to Mexico I could start receiving that financial boost much sooner. Thus I’m also protecting myself by keeping my options open to stay abroad (in the US for me) or return to Mexico. For example, I have a verbal agreement with my boss on the length of my current job, but technically I’m an employee at will and can be fired pretty easily.

I also applied because I have several friends that were applying or recently applied and helped me a lot navigate the application process¹. Also, you never know if the application process will change and become more complicated later on: though I hope that it will get easier!

But back to protection and security. My father has taught me that it’s our responsibility to cover our backs. Lets imagine that I have dependents. It would be irresponsible of me to miss out on the financial supplement because I didn’t send my documentation on time or because I forgot to send in the renewal documents: maybe it wouldn’t affect me, but it would affect them. Completing my application in time is under my control. What CONACyT cares about, their evaluation process, the clarity and reliability of their website are not.

Beyond SNI

This attitude goes beyond SNI. That is why I compete when I can for awards and scholarships that I think fit me, even if its a long shot. No one will come to you and say, “hey, your work is great, now you are a SNI member” (or any award). I believe that you have to keep working and make it hard for others to say no to your applications. You have to keep trying though because you won’t get them all. That was one of my messages in my recent CDSBMexico keynote.

Here are my slides for my #CDSBMexico remote talk tomorrow for day 1 of the "Latin American R/BioConductor Developers Workshop 2018". 70 slides for ~20 min, I can do this! 💪🏾 Right @jtleek @Shannon_E_Ellis?https://t.co/xKh2214YbM #rstats #teaching @Bioconductor @CDSBMexico pic.twitter.com/SULAoPbHeZ
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) July 29, 2018

Here are some of my recent applications:

You still have time to participate in this year's SciLifeLab prize! I wrote a blog post about it & shared my unsuccessful 2017 entry https://t.co/6y7DOP1m1a Got Qs? SciLifeLabPrize@aaas.org @sciencemagazine #academia #science Thx @jtleek for the last minute recommendation letter! pic.twitter.com/Yfyq7j9RTc
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) June 20, 2018

Check our #Bioinformatics Peer Prize III entry at https://t.co/WdggoD6KPh highlighting recount2 https://t.co/tK6iRfUT21 We appreciate the votes! Team submission w @AbhiNellore et al #RNAseq #rstats #reproducibility #scitwitter Video code https://t.co/J0nwUcV3O4 w ari @seankross
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) February 5, 2018

I just submitted my rstudio::conf(2019) diversity scholarship app https://t.co/nmZY7UJsiP @rstudio Text entries have a 1000 char limit!

I'm sharing mine https://t.co/q3wqbSuVbW in case it helps other @LIBDrstats @CDSBMexico & #rstats members at large.#tip Always ask for help!
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) August 22, 2018

I just submitted my application for the "Early Career Clinical Research Symbiont Award". If I get selected, then I'll be able to attend #PSB19 @PacSymBiocomp. Thx to someone who prompted me to send my application! You should submit yours too. Deadline 9/30 https://t.co/Pv1aHySVbI
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) September 28, 2018

So what happens now? I updated my CV as a friend suggested to list that I’m a SNI National Researcher Level 1.

Acknowledgments

I want to thank my friends that helped me during the SNI application process!

Gracias @sur_hp, @FSanchezQuinto, @mariagutierrez, @areyesq, @malacopa_genome, @mgschiavon, @AleMedinaRivera por el apoyo en todo el proceso de enviar la solicitud y motivarme a enviarla!

Ahora.... me falta obtener la FIEL para firmar el doc de @Conacyt_MX (creo)... ugh!
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) September 26, 2018

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
sessioninfo (Wickham, Chang, Flight, Müller et al., 2021)
knitcitations (Boettiger, 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, W. Chang, R. Flight, K. Müller, et al. sessioninfo: R Session Information. R package version 1.2.2. 2021. URL: https://CRAN.R-project.org/package=sessioninfo.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo   * 1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Actually, we are still trying to find out how we can sign the agreement from abroad. ↩︎

Problems with an article from the Baltimore Sun covering Dr. Mahoney-Steel’s immigration issues

Wed, 19 Sep 2018 00:00:00 +0000

Today, September 19th 2018, Dan Rodricks (Twitter: DanRodricks) published an article in the Baltimore Sun. The article was shared to me with the title I only thought this happened to Mexicans^[I saw the article as a photo that was shared by Hopkins colleagues. Dan Rodrick’s Facebook page also shows the article with the original title.] and is currently titled^[As of 09/19/2018 at 10 pm ET.] as “Rodricks: Hopkins library specialist hit by immigration crackdown after being blindsided by visa denial”.

![](http://lcolladotor.github.io/post/2018-09-19-problems-with-an-article-from-the-baltimore-sun-covering-dr-mahoney-steel-s-immigration-issues_files/Screen Shot 2018-09-19 at 11.43.50 PM.png){width=600px}

I didn’t like the title at all nor did I feel right when reading the article. I shared it with about 50 other Mexicans in Baltimore^[Many of us are students, alumni or even staff members at Hopkins and affiliated institutes (like myself).], talked about it over dinner, discussed a bit with Hopkins colleagues, and now thanks to all of them I have a clearer idea of what my problems with this article are. And there are many of them, but we can group them into:

the reporting by Dan Rodricks,
the portrayed handling of Dr. Mahoney-Steel’s case by Johns Hopkins University,
and the portrayed lack of attention by Dr. Mahoney-Steel.

Before I get into all of it, there is a petition online in support of Dr. Mahoney-Steel that you can read and sign here if you want: nearly 2,000 people have already signed it. You can also read other’s reasons for signing the petition. Right now I do not feel comfortable signing this petition, but maybe you are.

The reporting by Dan Rodricks

The article starts by framing Dr. Mahoney-Steel’s story around the fact that she had to return to England due to issues with her immigration paperwork and that:

the reaction of several friends and colleagues was uncannily the same: “I thought this only happened to Mexicans.”

There was no need at all to grab attention to Dr. Mahoney-Steel’s story by mentioning Mexicans or any other Spanish-speaking immigrants. Dan Rodricks is trying to get the reader to think that Mexicans are illegal immigrants (which many politicians have said), and thus that they aren’t welcome, and then ohh, others aren’t welcome either. Another interpretation of this quote is that Mexicans are uneducated, which would be equally offending.

While I’m glad that the title has been changed at the Baltimore Sun’s website, the first paragraph and last paragraph of the story still mention this quote that is attributed to friends and colleagues of Dr. Mahoney-Steel. They might have said this, but Dan Rodricks could have simply ignored their quote and framed his story along the lines of the updated title: a library specialist that is suffering because of the current immigration policies.

My second problem with the reporting is that there is a lack of information that is important in this context. There are many ways to become a permanent resident in the US and thus get a green card. Some of these ways involve getting sponsored by your employer. Some don’t. Basically, in some situations it would be the employer’s responsibility (Johns Hopkins University in this case) to be involved in and be supportive and in others I think that they are not obliged to help financially. You can have very different reactions to this story based on what mechanism Dr. Mahoney-Steel was asking Johns Hopkins University to help her.

The portrayed handling of Dr. Mahoney-Steel’s case by Hopkins

I’m using portrayed here because we currently don’t know Hopkins’ side.

What seems to be the main reason why Dr. Mahoney-Steel’s case became a story is because she submitted her documents to request a renewal of her H-1B (highly specialized worker) visa to Johns Hopkins University’s Office of International Services. She didn’t hear back from them until she heard in August 21st that they decided not to send her documents to the US government and that she should leave the country by August 31st.

This is hugely disturbing and would be enough to grab the attention of many readers. This is what caught the attention of several colleagues at Hopkins. We haven’t really heard the story from OIS’ perspective and I would definitely want to know more^[If they ever can share their story due to privacy restrictions.].

Due to the lack of details in the article, we don’t know exactly which mechanism Dr. Mahoney-Steel was asking Hopkins to help her with. She might have been considering applying under employed-based immigration via EB-1 or EB-2. For the EB-1 I think that your employer has to help you quite a bit. But not all employers want to do so. Also these processes take a considerable amount of time, not 10 days.

Alternatively, maybe Dr. Mahoney-Steel was asking Hopkins to help her pay for her green card application through family since she’s married to a US citizen and due to their portrayed extremely poor communication on her H-1B visa renewal. I haven’t heard of any employer helping their employees with the green card fees in such cases. It’d be great if they did it, but I wouldn’t expect this to happen.

The portrayed lack of attention by Dr. Mahoney-Steel

Again, we don’t know how everything exactly happened. My issues here could be based on how Dan Rodricks used his notes from interviewing Dr. Mahoney-Steel.

Many of the Mexicans I shared the article with are not US citizens^[You can be a dual citizen.] and thus have different types of visas that allow us to study and/or work in the US. We learn fast, either through personal experience or stories from others, that you have to prioritize any visa paperwork you have to do. That involves lots of planning and attention to detail. For example:

Knowing when your visa expires.
Learning the types of visas you can apply to after your current one expires.
Fees associated with the different visa applications.
Requirements for the visa you want to get: timeline, documents that you have, documents that you need to get, etc.
Communicating with all parties involved.

With this information at hand you can make plans. OIS is also pretty helpful, in my experience, as they provide seminars where they explain a lot of this information and you can also request to meet with one of their specialists. OIS also sends reminders that prompt you to get moving: get your documents ready and send them on time. Having defended OIS based on my experience I also have to say that I’ve heard them make costly mistakes that affected friends of mine. Ultimately, the immigrant is responsible for their own application and has to double check everything others make.

Additionally, I have learned myself that if something says that it can take between 60 and 90 days, that I should think that it’ll take 90 days and be happy if it takes less than that. This doesn’t mean that our plans are the best or that they always work out. It just means that we try to be prepared.

Having said all that, as portrayed, I’m surprised by Dr. Mahoney-Steel because:

She didn’t contact OIS a few days after sending her documents to follow up on her case. If she had, maybe she would have had more time to convince them to actually send her H-1B visa renewal documents to the US government, ask for help for her green card (EB-1, EB-2?, EB-2 NIW?), ask for her direct employer (Sheridan Libraries) to intervene, etc. Still, OIS should have immediately notified her that they weren’t going to send her H-1B documents to the US government!!
She has been married to a US citizen for less than two years (but over one)… and didn’t get a green card via an application through family. To my understanding, that’s one of the cheapest and fastest ways^[It will still take months, say about 11 to 13 months from what I’ve heard.] to apply for a green card. If you can get a green card, then you can stop worrying about visas. Isn’t that something we all want? Yes, it’s not free. The I-485^[The form you submit to get a green card.] fee is 1,225 USD for someone aged 14 to 78^[You might need to cover about another 400 USD in fees to get the documents that show that you have all the vaccines you need if you go to a place like Passport Health.]. Say that you are applying via EB2-NIW and hiring some lawyers, then you are looking at 5,000 USD or more in lawyer fees alone plus another 700 USD for the government. Given the large advantages that you get from having a green card and her husband’s situation, I would hope that many would have helped Dr. Mahoney-Steel raise 1,225 USD. Luckily for Dr. Mahoney-Steel, she can still apply for a green card through family even if she is living outside the US. I would hope that Johns Hopkins would be willing to hire her back then, or maybe even now and have her work remotely from the UK while the green card application is being processed (if Hopkins has a branch in Europe).

It does seem like Hopkins realized that they made a mistake (as portrayed in the article) because they covered her flight back to England.

I’m sorry to say this, but it ultimately looks to me that Dr. Mahoney-Steel felt entitled and never expected to face immigration problems. Though again, that’s the current lens I have, that is, Dan Rodricks’ article and the infamous quote from her friends and colleagues. That’s why it would be useful to hear more from Dr. Mahoney-Steel.

Closing remarks

I hope that Dan Rodricks and/or the Baltimore Sun edits his article to remove any mentions about Mexicans, which as they stand currently are all negative and ultimately offensive.

I also hope that we get some clarification from Johns Hopkins University Office of International Services. Depending on what happened, they might have to apologize to Dr. Mahoney-Steel.

If you are an immigrant in the US, I encourage you to plan ahead as much as you can anything related to visa paperwork. The current administration will make it as hard as they can to legally immigrate to the US, read for example this article: “Authorities Can Now Deny Visa and Green Card Applications Without Giving Applicants a Chance to Fix Errors”. But you don’t need to make it harder on yourself by lack of planning. Take advantage of all the resources you have at your employer (OIS in this case) and ask your fellow immigrants to share their knowledge with you. Double check every document because others (like your employer) might have made mistakes in a form or letter (basically don’t trust experts will do their job perfectly) because the US government will make you pay (in time, lost opportunities, etc) if you make a mistake. If you are thinking about getting a green card, you can start by checking websites like this one that lists all the possible ways to get one and by checking the USCIS website.

Finally, I’ll write a letter to the Baltimore Sun including a link to this post. But that’s for tomorrow September 20th 2018.

At least two friends thought of this quote by Martin Niemöller (which I didn’t know myself):

First they came for the socialists, and I did not speak out—because I was not a socialist.

Then they came for the trade unionists, and I did not speak out— because I was not a trade unionist.

Then they came for the Jews, and I did not speak out—because I was not a Jew.

Then they came for me—and there was no one left to speak for me.

Acknowledgments

I would like to thank those who I discussed this article with and corrected some of my facts. If you want me to name you let me know and I’ll edit this post. I didn’t because going the other way around (from named to anonymous) is challenging.

What about a lawyer-like app as the minimum help for defendants in immigration cases?

Mon, 17 Sep 2018 00:00:00 +0000

Today I attended the special panel discussion event at JHSPH called “Separated: Children Separation at the Border A Health and Human Rights Perspective”. It got my mind racing and here’s an idea. It’s likely (definitely) incomplete, but maybe it’ll get others to think on related ideas.

[Image source](http://hopkinshumanitarianhealth.org/news-events/events/separated-child-separation-at-the-border-a-health-and-human-rights-perspect)

Panel summary

The panel was composed by:

Colleen Kraft, President, American Academy of Pediatrics
Eric Schwartz, President, Refugee International
George Escobar, Chief of Program and Services, CASA de Maryland
Paul Spiegel, Director, Center for Humanitarian Health

I missed the first 30 minutes or so but I still got to listen to most of it. The panel members presented many facts and here are some that will be relevant to the idea I have:

Child separation is just one of the consequences of the current’s administration immigration policies implemented and enforced by the Department of Justice (ultimately headed by Jeff Sessions, the US Attorney General).
The US is the only country (to the panel’s members knowledge) with a child separation policy.
Due to empathy many individuals across political lines reacted against child separation.
The US immigration system won’t really change much even if Democrats get elected. Obama did deport over 2 million individuals, though he prioritized criminals¹.
Arrested immigrants are not required by law to have representation (a lawyer) provided by the government².
Minors (say 3 year old children) with no lawyers are being highlighted in the media.
Immigration cases where the defendants have lawyers drastically improve³ the odds for the defendants.
Immigration judges are human.
Immigration judges typically used to (or maybe still do) try to give time for a minor to get a lawyer.
Immigration judges are alledgely⁴ being pressured to meet quotas in the range of 700 to 1,000 cases by year under the current administration. Thus judges sometimes have to close cases in a couple of hours.
Immigration judges now basically have 2 options for closing a case: order deportation or (I’m missing the correct term) free the defendant.

At the end the panel members highlighted that we should take some type of action⁵ but that we should consider the consequences of our suggested policy changes. They also mentioned that we should take advantage of this moment (child separation got everyone’s attention) to raise the profile of the other problems with the current immigration policies.

My way of taking action: here’s an idea

I’m by no means an immigration expert. My way of taking action is to share ideas, like I’ve done in the past, that might be incomplete, unrealistic or even super flawed, but that hopefully motivate others.

The bare bones version of my idea was: what if immigrants could have an automatic (programmed) lawyer and translator during their hearings? This would not be a replacement for actually having lawyers (say those provided by local governments or NGOs as George Escobar mentioned) but would raise the minimum bar for those immigrants who currently have their cases processed with no lawyers at all.

Getting into the details

Imagine that we could get our hands on dozens/hundreds/thousands? of transcripts of immigration court hearings where we have the following information:

what the judge said
what the government’s lawyer said
what the defendant’s lawyer said (either to the judge or to the defendant)
what the defendant said

Just like a script for a play. We would additionally need a table with court hearing metadata such as:

Outcome: deportation, being freed (term?).
Date of the hearing.
State where the hearing occurred.

Then using machine learning (maybe with deep learning methods) process the text and try to determine potential suggestions an actual defendant’s lawyer would give to its defendant or respond to the judge/government’s lawyer. It might not always get things right, but I imagine that it would be better than the current state of affairs.

The automatic lawyer would need then to work as say a phone app that listens to what others are saying in the room. Say have 3 icons with one per person present (judge, defendant, gov’s lawyer). Then the defendant presses each button when each person is talking. The app then shows some 1 to say 3 suggested responses (with translations)⁶ and responds for the defendant in English once the defendant chooses an option (or goes with the top one).

Implementing an initial version of the app

I think that large computing companies like Amazon, Microsoft and Google would be willing to provide some compute credits on their clouds for the initial version of the algorithm that is listening to the court hearing and then provides suggestions. Think of this as the “suggested text” you get nowadays when typing emails on Gmail or text messages. Maybe these companies have programs where you can apply to have some of their engineers help you for a certain number of hours.

I think that one big initial challenge would be to get that collection of transcripts from immigration court hearings where defendant lawyer(s) were present.

I also imagine that automated lawyers are not allowed currently in courts. But compared to providing human lawyers in all immigration cases, this change might be more realistic to pass as a law.

I also imagine that some phone companies might be willing to provide some refurbished phones that only have this app installed and are kept safe in the immigration courts. And well, satisfy any security requirements the government has.

Improving the app

Lets say that you get that alpha version of the app working. If we had volunteer lawyers annotate the transcripts with some information about the intent behind what each person said (it could start with just 3 options: negative, positive, neutral from the perspective of the defendant) that could maybe help the algorithm that processes the transcripts.

If we also had more detailed court hearing metadata such as:

Location of the court (I imagine a that a few judges work in each court)
Demographics of the defendants: like which country or even region of the country where they come from, whether the defendant has any family support, etc
Category information for the court hearing (maybe cases can be grouped into a few categories)

then I imagine that the app would be able to have more personalized experience, like re-adjust the suggestions based on which court you are located at and adapt the translations to the Spanish version the defendant is most familiar with (we say buddy in so many different ways as shown below).

Image source

The app could also be connected to remote human immigration lawyers than can intervene remotely when the suggestions algorithm doensn’t know what to do. Maybe this could be part of some social service that immigration lawyers (regardless of their personal political preferences) could do as some elective during their formation.

Doing some convincing

I think that you could try to get support for this automatic immigration lawyer by arguing that:

It’s better than no lawyer.
It’s cheaper than having lawyers for all cases.
Improves processing times on average (ideally) such that immigration judges can fulfill their quotas (here I’m hoping that it reduces the rate at which cases are closed with deportation orders).
Is more humane than having a minor with no help. Though I hope that immigration judges would still try to give time for children to secure a lawyer.
Might be implemented earlier (years earlier?) than changes in immigration law requiring that all defendants have a human lawyer in immigration cases.
If human lawyers can help through the app, then it also increases the number of jobs for immigration lawyers.

This idea has potentially many flaws because like any system, people on both sides will try to game it. This is where having a large set of transcripts would be useful as well as continuous updates to the algorithm such that gaming the system actually becomes hard.

As you can see, this is just an idea, or a collection of them around one theme. It would need serious work to implement.

Acknowledgments

Do you want to listen to the whole panel discussion? From this tweet it looks like the recording will be available online:

Thank you @ColleenKraft (@AmerAcadPeds), @EricSchwartzRI (@RefugeesIntl), George Escobar (CASA de Maryland) & @pbspiegel (@Humanit_Health) for participating in today's panel. A recorded version of the talk will be available on the @Humanit_Health website: https://t.co/nXAvI5KJkm
— Johns Hopkins Bloomberg School of Public Health (@JohnsHopkinsSPH) September 17, 2018

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
devtools (Wickham, Hester, Chang, and Bryan, 2022)
knitcitations (Boettiger, 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, J. Hester, W. Chang, and J. Bryan. devtools: Tools to Make Developing R Packages Easier. R package version 2.4.5. 2022. URL: https://CRAN.R-project.org/package=devtools.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  callr           3.7.3      2022-11-02 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  crayon          1.5.2      2022-09-29 [1] CRAN (R 4.3.0)
##  devtools      * 2.4.5      2022-10-11 [1] CRAN (R 4.3.0)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  ellipsis        0.3.2      2021-04-29 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  fs              1.6.2      2023-04-25 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  htmlwidgets     1.6.2      2023-03-17 [1] CRAN (R 4.3.0)
##  httpuv          1.6.11     2023-05-11 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  later           1.3.1      2023-05-02 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  memoise         2.0.1      2021-11-26 [1] CRAN (R 4.3.0)
##  mime            0.12       2021-09-28 [1] CRAN (R 4.3.0)
##  miniUI          0.1.1.1    2018-05-18 [1] CRAN (R 4.3.0)
##  pkgbuild        1.4.2      2023-06-26 [1] CRAN (R 4.3.0)
##  pkgload         1.3.2      2022-11-16 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  prettyunits     1.1.1      2020-01-24 [1] CRAN (R 4.3.0)
##  processx        3.8.2      2023-06-30 [1] CRAN (R 4.3.0)
##  profvis         0.3.8      2023-05-02 [1] CRAN (R 4.3.0)
##  promises        1.2.0.1    2021-02-11 [1] CRAN (R 4.3.0)
##  ps              1.7.5      2023-04-18 [1] CRAN (R 4.3.0)
##  purrr           1.0.1      2023-01-10 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  remotes         2.4.2      2021-11-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo     1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  shiny           1.7.4      2022-12-15 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  urlchecker      1.0.1      2021-11-30 [1] CRAN (R 4.3.0)
##  usethis       * 2.2.1      2023-06-23 [1] CRAN (R 4.3.0)
##  vctrs           0.6.3      2023-06-14 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  xtable          1.8-4      2019-04-21 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

George Escobar mentioned that they protested the fact that a DUI labeled someone as a criminal, but well, Trump has gone beyond DUIs arrested many individuals with no criminal history. ↩︎
George Escobar highlighted that one path of action is to convince local governments to fund/provide lawyers to these individuals. ↩︎
I don’t know by how much. ↩︎
Information has leaked about this but I guess that it’s not completely public info. ↩︎
Could be with your vote, helping members in your community that are affected, improving how we translate research into terms everyone understands, improving the education about the violent reality many are trying to escape by coming to the US, approaching local governments, etc. ↩︎
The app could show the text but also should read the translations since very young children will very likely not know how to read. I think that the app should never assume that the defendant knows how to read. ↩︎

R Gene Regulatory Interaction Formulator For Inquiring Networks

Thu, 13 Sep 2018 00:00:00 +0000

This blog post was written by ME Martinez-Sanchez, S Muñoz, M Carrillo, E Azpeitia, D Rosenblueth and originally posted at the CDSB blog.¹

In this blog post we will describe the package rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) that was one of the projects developed during the TIB2018-BCDW. We hope to continue developing Griffin and rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019). If you have ideas, suggestions or bugs, please contact us via rGriffin GitHub repo.

The problem

Boolean networks allow us to give a mechanistic explanation to how cell types emerge from regulatory networks. However, inferring the regulatory network and its functions is complex problem, as the available information is often incomplete. rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) uses available biological information (regulatory interactions, cell types, mutants) codified as a set of restrictions and returns the Boolean Networks that satisfy that restrictions. This Boolean networks can then be used to study the biological system.

The rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) package is an R connector to Griffin (Gene Regulatory Interaction Formulator For Inquiring Networks), a java library for inference and analysis of Boolean Network models. Griffin takes as inputs biologically meaningful constraints and turns them into a symbolic representation. Using a SAT engine, Griffin explores the Boolean Network search space, finding all satisfying assignments that are compatible with the specified constraints. The rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) package includes a number of functions to interact with the BoolNet package.

A small example

Let us suppose a cell, we know that this cell has three proteins called a, b and c. We know that a activates b and that b and c inhibit each other. We also suspect that b and c may have positive self-regulatory loops. We can add this interactions to the table as “OPU” (optional, positive, unambiguous). This dataframe is the topology of the network.

Source	Target	Interaction
a	b	+
b	c	-
c	b	-
b	b	OPU
c	c	OPU

Suppose we also have some information of what cell types have been observed. For example, there is a cell type that expresses b, but not a or c and an other cell type that expresses c, but not a or b. There might exist a third cell type that has not been fully characterized where we know that the cell expresses no a or c but we have NO information on b. This dataframe is the attractors of the network.

a	b	c
0	1	0
0	0	1
0	*	0

We can then use this information to create a query. rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) can include other types of information like transition between cell type, cycles, transitions between cell types or mutant cell types.

## Install using:
# devtools::install_github('mar-esther23/rgriffin')
## Note that the package depends on rJava

library('rGriffin')

## Loading required package: rJava

genes = c('a','b','c')
inter = data.frame(source=c('a','b','c', 'b','c'), 
                  target=c('b', 'c', 'b', 'b', 'c'), 
                  type=c('+','-','-','OPU','OPU'),
                    stringsAsFactors = F )
q = create.gquery.graph(inter, genes)
attr = data.frame(a=c(0,'*',0), 
                 b=c(0,1,0), 
                 c=c(0,0,1),
                 stringsAsFactors = F )
q = add.gquery.attractors(q, attr)

Then we can use Griffin to find the networks that behave according with our biological information.

nets = run.gquery(q)
nets

## [1] "targets,factors\na,false\nb,((((!a&b)&!c)|((a&!b)&!c))|((a&b)&!c))\nc,(!b&c)\n"                         
## [2] "targets,factors\na,false\nb,(((((!a&b)&!c)|((a&!b)&!c))|((a&b)&!c))|((a&b)&c))\nc,(!b&c)\n"             
## [3] "targets,factors\na,false\nb,((((((!a&b)&!c)|((!a&b)&c))|((a&!b)&!c))|((a&b)&!c))|((a&b)&c))\nc,(!b&c)\n"
## [4] "targets,factors\na,false\nb,((((!a&b)&!c)|((a&b)&!c))|((a&b)&c))\nc,(!b&c)\n"                           
## [5] "targets,factors\na,false\nb,((((((!a&b)&!c)|((a&!b)&!c))|((a&!b)&c))|((a&b)&!c))|((a&b)&c))\nc,(!b&c)\n"

There are multiple options to integrate BoolNet and rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019). The function get.net.topology() can obtain the topology with interaction signs of a BoolNet network. The function attractor2dataframe() can be used to export a BoolNet attractor as a dataframe that rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) can use. The function run.gquery() includes the option return=’BoolNet’, that return the inferred networks as BoolNet networks.

History

The development of Griffin began in 2013 as a PAPIIT (Programa de Apoyo a Proyectos de Investigación e Innovación Tecnológica) project to solve the inference of Boolean Network models for the Arabidopsis thaliana root stem cell niche. It continued in 2015 with support of Conacyt grant 221341.

In January, 2017 we organized a course in C3-UNAM to teach biologist how to use Griffin. We received two main comments: the input format was too complicated and it was uncomfortable to use the output with other packages. After some consideration we decided to create an R wrapper that could export and import BoolNet networks. We selected BoolNet as it has an good documentation and the package BoolFilter had been designed to work with it.

The development of rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) began during the EOBM 2017 in CUIB. For the following year we continued developing rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) as our schedules allowed. There were multiple challenges during the development: defining user-friendly inputs, using Rjava, and structuring the package. In August 2018, we attended the TIB2018-BCDW where we received valuable guidance from Martin Morgan and Benilton S Carvalho. It was during this workshop that the first version of rGriffin (Martinez-Sanchez, Muñoz, Carrillo, Azpeitia, et al., 2019) was finished.

Acknowledgments

Here you can find the tweet about the original blog post:

Check our first contributed blog post: R Gene Regulatory Interaction Formulator For Inquiring Networkshttps://t.co/9azbElGZGH

ME Martinez-Sanchez, S Muñoz, M Carrillo, E Azpeitia, D Rosenblueth

Thx @mt_morgan @benilton!#CDSBMexico #TIB2018 #LatAmBioc18 #rstats #TIB #blog pic.twitter.com/TbJv37fkWQ
— ComunidadBioInfo (@CDSBMexico) August 21, 2018

This blog post was made possible thanks to:

References

[1] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.8. 2017. URL: https://CRAN.R-project.org/package=knitcitations.

[2] M. Martinez-Sanchez, S. Muñoz, M. Carrillo, E. Azpeitia, et al. rGriffin: Gene Regulatory Interaction Formulator For Inquiring Networks. R package version 0.1. 2019.

[3] H. Wickham, J. Hester, and W. Chang. devtools: Tools to Make Developing R Packages Easier. R package version 2.0.2. 2019. URL: https://CRAN.R-project.org/package=devtools.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. ISBN 978-0815363729. Boca Raton, Florida: Chapman and Hall/CRC, 2017. URL: https://github.com/rstudio/blogdown.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                                      
##  version  R version 3.5.2 Patched (2019-02-17 r76113)
##  os       macOS Mojave 10.14.4                       
##  system   x86_64, darwin15.6.0                       
##  ui       X11                                        
##  language (EN)                                       
##  collate  en_US.UTF-8                                
##  ctype    en_US.UTF-8                                
##  tz       America/New_York                           
##  date     2019-04-10                                 
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version date       lib source                            
##  assertthat      0.2.1   2019-03-21 [1] CRAN (R 3.5.2)                    
##  backports       1.1.3   2018-12-14 [1] CRAN (R 3.5.0)                    
##  bibtex          0.4.2   2017-06-30 [1] CRAN (R 3.5.0)                    
##  BiocManager     1.30.4  2018-11-13 [1] CRAN (R 3.5.0)                    
##  BiocStyle     * 2.10.0  2018-10-30 [1] Bioconductor                      
##  blogdown        0.11.1  2019-04-05 [1] Github (rstudio/blogdown@5179caf) 
##  bookdown        0.9     2018-12-21 [1] CRAN (R 3.5.0)                    
##  callr           3.2.0   2019-03-15 [1] CRAN (R 3.5.2)                    
##  cli             1.1.0   2019-03-19 [1] CRAN (R 3.5.2)                    
##  colorout      * 1.2-0   2019-02-18 [1] Github (jalvesaq/colorout@cc5fbfa)
##  crayon          1.3.4   2017-09-16 [1] CRAN (R 3.5.0)                    
##  desc            1.2.0   2018-05-01 [1] CRAN (R 3.5.0)                    
##  devtools      * 2.0.2   2019-04-08 [1] CRAN (R 3.5.2)                    
##  digest          0.6.18  2018-10-10 [1] CRAN (R 3.5.0)                    
##  evaluate        0.13    2019-02-12 [1] CRAN (R 3.5.2)                    
##  fs              1.2.7   2019-03-19 [1] CRAN (R 3.5.2)                    
##  glue            1.3.1   2019-03-12 [1] CRAN (R 3.5.2)                    
##  htmltools       0.3.6   2017-04-28 [1] CRAN (R 3.5.0)                    
##  httr            1.4.0   2018-12-11 [1] CRAN (R 3.5.0)                    
##  jsonlite        1.6     2018-12-07 [1] CRAN (R 3.5.0)                    
##  knitcitations * 1.0.8   2017-07-04 [1] CRAN (R 3.5.0)                    
##  knitr           1.22    2019-03-08 [1] CRAN (R 3.5.2)                    
##  lubridate       1.7.4   2018-04-11 [1] CRAN (R 3.5.0)                    
##  magrittr        1.5     2014-11-22 [1] CRAN (R 3.5.0)                    
##  memoise         1.1.0   2017-04-21 [1] CRAN (R 3.5.0)                    
##  pkgbuild        1.0.3   2019-03-20 [1] CRAN (R 3.5.2)                    
##  pkgload         1.0.2   2018-10-29 [1] CRAN (R 3.5.0)                    
##  plyr            1.8.4   2016-06-08 [1] CRAN (R 3.5.0)                    
##  prettyunits     1.0.2   2015-07-13 [1] CRAN (R 3.5.0)                    
##  processx        3.3.0   2019-03-10 [1] CRAN (R 3.5.2)                    
##  ps              1.3.0   2018-12-21 [1] CRAN (R 3.5.0)                    
##  R6              2.4.0   2019-02-14 [1] CRAN (R 3.5.2)                    
##  Rcpp            1.0.1   2019-03-17 [1] CRAN (R 3.5.2)                    
##  RefManageR      1.2.12  2019-04-03 [1] CRAN (R 3.5.2)                    
##  remotes         2.0.3   2019-04-09 [1] CRAN (R 3.5.2)                    
##  rGriffin      * 0.1     2019-04-10 [1] local                             
##  rJava         * 0.9-11  2019-03-29 [1] CRAN (R 3.5.2)                    
##  rlang           0.3.4   2019-04-07 [1] CRAN (R 3.5.2)                    
##  rmarkdown       1.12    2019-03-14 [1] CRAN (R 3.5.2)                    
##  rprojroot       1.3-2   2018-01-03 [1] CRAN (R 3.5.0)                    
##  sessioninfo     1.1.1   2018-11-05 [1] CRAN (R 3.5.0)                    
##  stringi         1.4.3   2019-03-12 [1] CRAN (R 3.5.2)                    
##  stringr         1.4.0   2019-02-10 [1] CRAN (R 3.5.2)                    
##  testthat        2.0.1   2018-10-13 [1] CRAN (R 3.5.0)                    
##  usethis       * 1.5.0   2019-04-07 [1] CRAN (R 3.5.2)                    
##  withr           2.1.2   2018-03-15 [1] CRAN (R 3.5.0)                    
##  xfun            0.6     2019-04-02 [1] CRAN (R 3.5.2)                    
##  xml2            1.2.0   2018-01-24 [1] CRAN (R 3.5.0)                    
##  yaml            2.2.0   2018-07-25 [1] CRAN (R 3.5.0)                    
## 
## [1] /Library/Frameworks/R.framework/Versions/3.5/Resources/library

I’m re-posting their blog post here with their permission so that their post will be shared more widely via R-Bloggers and R Weekly among other sites since the CDSB blog is currently not linked to those aggregators. The original blog post has been edited so that it evaluates the R code and includes all the information needed for reproducibility.↩

Harassment, diversity in science and inspiration from my grandmother

Mon, 30 Jul 2018 00:00:00 +0000

Today was a big day. I care about many things including diversity in science (STEM) and building a community of R users and developers in Mexico. Both moved forward in two completely separate conferences: one in Mexico: CDSBMexico; and one in Canada: JSM2018.

CDSBMexico

This was a very important day for me. It was the beginning of the Latin American R/BioConductor Developers Workshop 2018 in Cuernavaca, Mexico. I already wrote a blog post about why I was super excited about CDSBMexico, but briefly it’s because this is something we’ve been wanting to see become a reality for years and have been working towards it.

Here are my slides for my #CDSBMexico remote talk tomorrow for day 1 of the "Latin American R/BioConductor Developers Workshop 2018". 70 slides for ~20 min, I can do this! 💪🏾 Right @jtleek @Shannon_E_Ellis?https://t.co/xKh2214YbM #rstats #teaching @Bioconductor @CDSBMexico pic.twitter.com/SULAoPbHeZ
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) July 29, 2018

My job was to give the first keynote to pump up everyone at the workshop. So I had a bit of trouble sleeping and woke up remembering a dream. If you know me, I never dream: or never remember my dreams. It was a dream of my grandmother Mercedes Vides Tovar.

Grandma

I have many things from my grandparents in me, like Juan’s joviality, Rosa’s love, Rolando’s tenacity to overcome tough moments. Or so I like to think. But my grandma, Abue, has always been an academic inspiration. She overcame many challenges in her life. She studied at the university to become a doctor, even without total family support (to avoid details). She was a trailblazer in the field of Epidemiology. She had to flee her native country leaving everything behind. She studied at the Institut Pasteur in Paris, France while caring for 3 adolescents as a single mom. She studied in a time when WomenInSTEM had to disguise themselves up as men and were the targets of jokes all the time. Rolando, her classmate at the university, has told me a handful of these jokes: I wouldn’t be the only one to call many of them harassment. She left academia to start her own business in public health. She was the main health supervisor in the first international conference my dad organized early in his career (which helped him launch it): I was the 6 year old in charge of delivering the gifts in the closing ceremony ^^. She would answer my phone calls when I first started French school and my dad was busy at work: thanks to their time in Paris my dad fell in love with the French way of teaching math.

I actually don’t know much more. She passed away when I was 13 after a years long battle with disease. Google tells me that she is a co-author of at least three titles in the field of Public Health:

1968 Organización del servicio social de los pasantes de Odontología en México
1970 Situación epidemiológica de la frontera norte de México y programa de vigilancia
1972 Proyecto de organización y funcionamiento de los laboratorios de salud pública en los estados de la frontera norte: also, she’s the sole author.

I did inherit her souvenirs from her trips (my dad also loves them) and something that is precious to me: a medal with her name. I’ve looked at that medal next to my bed in Mexico many times. I would look at it, search for inspiration, then try harder to solve whatever academic problem I had.

In my dream there was a moment where I was sitting next to her (me in my current adult form) and we were revising many of our shared moments. She was explaining to me things she couldn’t explain before because I wouldn’t have understood them. She also pumped me up before my day and wished me luck for my talk. I think that I delivered ^^

@CDSBMexico @fellgernon dio una charla inspiradora <3 #collaboration #rstats #TIBs2018 @nnb_unam
— Semiramis (@semiramis_cj) July 30, 2018

WomenInStem

Today was a big day for WomenInSTEM and in particular in the field of Statistics. Why? Today the Joint Statistical Meetings (JSM2018) held a session on “Addressing Sexual Misconduct in Statistics” that was organized by Stephanie Hicks and chaired by Keegan Korthauer.

For #JSM2018, I organized a session on Addressing Sexual Misconduct in Statistics. If you plan to attend the meeting, I invite you to join us https://t.co/UxpUiqPzTy pic.twitter.com/IdShEPlqi6
— Stephanie Hicks (@stephaniehicks.bsky.social) (@stephaniehicks) June 6, 2018

The session is not a direct response (I think) but it’s definitely related to Kristian Lum’s blog post titled Statistics, we have a problem and to the more general MeToo movement.

Sexual harassment and misconduct not a US problem only. We have a code of conduct at CDSBMexico both in English and Spanish to help prevent any issues.

Q by me: what to do when people outside the U.S. say that this is an American problem and they don't need to have a code of conduct for their meeting? #JSM2018
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) July 30, 2018

I couldn’t be there, but I wasn’t the only one excited following what has happening online thanks to tweets from many including Mandy Mejia and Michael Hoffman.

THIS IS ACTUALLY HAPPENING. Now in CC-West Ballroom A. #JSM2018 pic.twitter.com/Pw9zdlhi02
— Mandy Mejia, PhD (@mandyfmejia) July 30, 2018

Times are changing

I do think that slowly, maybe sometimes quickly, the times are changing. I’ve been telling friends and family for a few years now that I’ve come to realize that we need to keep trying to improve, to stay updated, to do what’s best and maybe sometimes come up with new ideas to improve things. I do care quite a bit for WomenInSTEM but for me, it’s larger than that. It’s about diversity. I don’t know in how many different ways we can label each other and it’s not the point. So yes, gender diversity is a good thing and initiatives like this seem like a good thing to do:

Looks like a good thing to do. https://t.co/m2sjLYB1Er #ashg18 pic.twitter.com/iKofjrIKTx
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) July 30, 2018

But then I think immediately of next steps, like race and ethnicity as well as sexual diversity. I want to ask for more, but I sometimes hold myself back. I’m no expert on many of these topics, so maybe what seems like a good policy could easily backfire later. So I think about incremental steps. But maybe that’s too slow and not enough. Maybe careers are suffering but I guess that as a community we are going as safely and fast as we can.

This 💯! My career trajectory would have been completely different in absence of hostile environments. https://t.co/aqmQtVyXUW
— Dr. Elana J Fertig (@FertigLab) July 30, 2018

I’m not supporting whoever Michael was referencing in his next tweet. I don’t flirt at conferences: I do socialize and try to make new friends with fellow conference attendees. I do acknowledge that I struggle in my head with what is ok and not ok to do, particularly with people you spend at lot of time with at school, work, etc. I’m not certain of the boundary between clumsy dating skills and sexual misconduct in some scenarios. I think that I’m not the only one because I’ve heard different versions in different sexual harassment prevention sessions. So mostly I end up doing nothing yet sometimes I wonder if I lost out because I didn’t try a bit more with X person. For instance, Mercedes maried her classmate Rolando. What can I say, I’m still learning and we all probably need to learn to differentiate these boundaries better through more training.

We ran out of time so we didn't get to hear the question from the dude who was saying that something (I presume flirting at conferences) was "human nature". Our loss, I'm sure #JSM2018
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) July 30, 2018

I also know that sometimes I’ve been a bystander and failed to take action. So this next idea sounds good to me. I do try to intervene more now, specially with micro aggressions: because they are less high pressure situations and easier to do something about, at least that’s how I feel.

Bystander training might help. It's not easy to do the right thing in a difficult situation when you have no idea what the right thing is, no training, no experience, no practice. #JSM2018
— Michael Hoffman @michaelhoffman.bsky.social (@michaelhoffman) July 30, 2018

To close my post, I encourage you to check their request for feedback!

If you’re planning to come to our session (211) on sexual misconduct today (and even if you’re not) you might take a look at these scenarios that we’ll use to get the conversation started. Link in RT below. #JSM2018 https://t.co/czjUKYkHq1
— Kristian Lum (@KLdivergence) July 30, 2018

And if you are an R non-cis male person, you should totally join the R-Ladies community slack!

📢 Join our new ✨ R-Ladies community Slack! ✨

It's aiming for a safe & global space to discuss topics and share ideas around #rstats & the #rladies community! 💜

We invite all non-cis male R users to sign-up via https://t.co/TOXALUGLuf 🚀 pic.twitter.com/ZMNuqkN0HW
— R-Ladies Global (@RLadiesGlobal) July 25, 2018

Acknowledgments

This blog post was made possible thanks to:

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[2\] H. Wickham, J. Hester, W. Chang, and J. Bryan. devtools: Tools to Make Developing R Packages Easier. R package version 2.4.5. 2022. URL: https://CRAN.R-project.org/package=devtools.

\[3\] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  callr           3.7.3      2022-11-02 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  crayon          1.5.2      2022-09-29 [1] CRAN (R 4.3.0)
##  devtools      * 2.4.5      2022-10-11 [1] CRAN (R 4.3.0)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  ellipsis        0.3.2      2021-04-29 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  fs              1.6.2      2023-04-25 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  htmlwidgets     1.6.2      2023-03-17 [1] CRAN (R 4.3.0)
##  httpuv          1.6.11     2023-05-11 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  later           1.3.1      2023-05-02 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  memoise         2.0.1      2021-11-26 [1] CRAN (R 4.3.0)
##  mime            0.12       2021-09-28 [1] CRAN (R 4.3.0)
##  miniUI          0.1.1.1    2018-05-18 [1] CRAN (R 4.3.0)
##  pkgbuild        1.4.2      2023-06-26 [1] CRAN (R 4.3.0)
##  pkgload         1.3.2      2022-11-16 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  prettyunits     1.1.1      2020-01-24 [1] CRAN (R 4.3.0)
##  processx        3.8.2      2023-06-30 [1] CRAN (R 4.3.0)
##  profvis         0.3.8      2023-05-02 [1] CRAN (R 4.3.0)
##  promises        1.2.0.1    2021-02-11 [1] CRAN (R 4.3.0)
##  ps              1.7.5      2023-04-18 [1] CRAN (R 4.3.0)
##  purrr           1.0.1      2023-01-10 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  remotes         2.4.2      2021-11-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo     1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  shiny           1.7.4      2022-12-15 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  urlchecker      1.0.1      2021-11-30 [1] CRAN (R 4.3.0)
##  usethis       * 2.2.1      2023-06-23 [1] CRAN (R 4.3.0)
##  vctrs           0.6.3      2023-06-14 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  xtable          1.8-4      2019-04-21 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

SciLifeLab Prize: you still have time to participate!

Wed, 20 Jun 2018 00:00:00 +0000

I have recently been getting reminder emails from the Science & SciLifeLab Prize for Young Scientists. The application deadline is July 15th, 2018!

Last year I submitted an entry to this competition and I enjoyed the experience, even if it was a bit rushed. The process of joining the competition is relatively straight forward:

Write an essay about your Ph.D. thesis work.
Get a recommendation letter from your Ph.D. advisor.
Write a short Ph.D. thesis abstract section, list your affiliation, education, academic and professional awards and professional experience.
Include references.

You don’t need to pay for competing! You already did the very hard part of completing your Ph.D.!!! I also like competitions like this one that invite you to think about your thesis and how to explain it. You should note that you are only eligible to participate once in this competition.

You can read about the 4 winners from 2017 at www.sciencemag.org/prizes/SciLifeLab/2017 and other years too to draw some inspiration from. I didn’t win, but I’m still going to share my entry in case it’s useful to others. I’m also sharing it to keep with the theme of also sharing your failures that I talked in my mindfulness post.

I don’t know exactly why my entry was not selected (you might have some ideas on how to improve it!). But I can tell you that:

I knew the deadline was coming but I wrote it the week it was due.
I didn’t have time to ask for some more specific feedback from others.
I forgot that a recommendation letter was needed, so I asked at the very last minute for one by Jeff Leek. I don’t know what magic he did (likely skipping a few hours of sleep, sorry!!!) but he managed to send it in time. Oops!

I’m not sure if I should have emphasized more the impact of my Ph.D. thesis work or what else to do. However, I do know that this exercise was useful. First, just re-reading my essay makes me smile. Also, Figure 1 shown here became a figure in the recount-workflow paper (Collado-Torres, Nellore, and Jaffe, 2017). Furthermore, when I was notified about the Thinkable Bioinformatics Peer Prize III I didn’t have to think twice about joining that competition. Finally, since I wrote it in an Rmd file, it was easy to modify for this blog post.

Like I’ve said, you have nothing to lose by joining this competition. Yes, you have to invest a bit of time. But it can be helpful and morale boosting! Plus, you have a chance to win it!!! Just remember that the deadline is July 15th, 2018 (and to give your advisor enough time to write their recommendation letter for you)!

In case you are curious, here are my full entry files (the figure is here though).

\[1\] L. Collado-Torres, A. Nellore, and A. E. Jaffe. “recount workflow: Accessing over 70,000 human RNA-seq samples with Bioconductor \[version 1; referees: 1 approved, 2 approved with reservations\]”. In: F1000Research (2017). DOI: 10.12688/f1000research.12223.1. URL: https://f1000research.com/articles/6-1558/v1.

Without further ado, here’s my full entry:

Usable human gene expression data and annotation-agnostic methods

In the last decade RNA sequencing (RNA-seq) has become the predominant assay for measuring gene expression. RNA-seq allows us to measure all expressed genes, improve gene and transcript annotation, and measure expressed sequences that otherwise are excluded in microarray studies. Typical RNA-seq analysis starts with a matrix containing the number of RNA-seq reads overlapping each gene for each sample (Michael I. Love et al. 2016; Law et al. 2016). To compute such a matrix, you first map the raw data to the genome with aligners such as TopHat2 and HISAT (Kim et al. 2013; Kim, Langmead, and Salzberg 2015) and then use tools such as Rsubread and HTSeq (Y, GK, and W 2013; Anders, Pyl, and Huber 2014) to construct the read count matrix. After measuring enough samples, you can determine which genes are differentially expressed between two or more groups.

When I started my graduate studies with Jeff Leek, the most commonly used methods for differential expression analysis were DESeq (Anders and Huber 2010) and edgeR (Robinson, McCarthy, and Smyth 2010). Leek and colleagues put together a set of gene count matrices in a project called ReCount (Alyssa C. Frazee, Langmead, and Leek 2011). ReCount allowed researchers to access several datasets without having to run the whole processing pipeline. ReCount then helped the development of new methods such as DESeq2 (Michael I. Love, Huber, and Anders 2014). It was also Leek and colleagues that decided to look at RNA-seq data in a manner less dependent on potentially incomplete gene annotation. With the increase in size of RNA-seq projects, they thought it would be feasible to assess the differential expression signal at base pair resolution in an approach called differentially expressed regions (DER) finder (Alyssa C. Frazee et al. 2014).

It was around then that I started to work with Andrew Jaffe, who had RNA-seq data from the human brain. Jaffe and colleagues hypothesized that the gene annotation for the human brain might be incomplete and were interested in applying the DER finder approach to their data. With them, I modified the DER finder approach to make it computationally feasible to analyze these human brain samples at base resolution (Jaffe et al. 2015). We indeed found that the human brain transcriptome was not fully annotated and identified intronic DERs that were present in other brain datasets, including from mouse, and showed these regions were enriched for genetic risk for brain disorders. Further computational improvements to the DER finder approach yielded the derfinder (Collado-Torres, Nellore, Frazee, et al. 2017) Bioconductor (Huber et al. 2015) package, reducing computing time from days to hours.

While I was working on derfinder, Abhinav Nellore was developing Rail-RNA (Nellore, Collado-Torres, et al. 2016) with the goal of aligning large RNA-seq datasets. We determined that we only needed coverage data instead of alignments to use derfinder and to perform gene level analyses. Coverage data files are much smaller than alignment files, which meant that we could greatly reduce our storage costs when using Rail-RNA and allowed us to think as big as possible. Initially we looked at 21,504 human RNA-seq samples and explored the landscape of exon-exon junctions (Nellore, Jaffe, et al. 2016). The reads spanning exon-exon junctions provided by Rail-RNA can be used for a second type of annotation-agnostic analysis as these reads provide information about exon skipping, alternative donor/acceptor sites and novel events. With a fairly conservative filter we determined that 18.6% of exon-exon junctions were missing in the annotation (Nellore, Jaffe, et al. 2016). This observation along with the fact that exon-exon junctions can be used to determine differential transcript usage gave strength to this second annotation-agnostic RNA-seq analysis method.

Researchers around the world have shared their raw data via the Sequence Read Archive or via large consortiums such as the Genotype-Tissue Expression study (GTEx) and the Cancer Genome Atlas (TCGA). However, it still is complicated to align and compute count matrices for these datasets. We thought that if the public data was uniformly processed that we could make this large body of human expression data more reusable. We were well positioned to carry out this project and just needed resources. Luckily, Jeff Leek, Ben Langmead and others got together and funded the implementation of Rail-RNA to analyze all the human public RNA-seq data available at the time, over 70,000 samples.

Figure 1: Overview of the data available in recount2. Reads aligned to the reference genome can be used to compute a base pair coverage curve, identify exon-exon junctions and compute gene and exon count matrices. The reads spanning exon-exon junctions (jx) are used to compute a third count matrix that includes un-annotated junctions (jx 3 and 4). Annotation-agnostic expressed regions can be determined from the coverage data.

We computed gene count matrices for these 70,000 samples and made them available as the recount2 resource that can be accessed at jhubiostatistics.shinyapps.io/recount/ as well as through the recount (Collado-Torres, Nellore, Kammers, et al. 2017) Bioconductor package, thus representing a major improvement over ReCount. To take advantage of the data as much as possible, we also released exon and exon-exon junction count matrices as well as the coverage data files required for the DER finder approach, thus enabling both annotation-agnostic methods we developed that complement traditional methods. Figure 1 illustrates the type of information that is available via recount2.

My colleagues and I believe that recount2 will fuel the development of new analytical methods and greatly increase the usability of the public human gene expression data researchers have collected over the years. In the meantime, we are working hard on improving the data in recount2 (Ellis, Collado-Torres, and Leek 2017; Wilks et al. 2017) and exploring how to leverage results from different RNA-seq analytical approaches (Jaffe et al. 2017).

Ph.D. thesis abstract

Leonardo Collado-Torres’ thesis work is centered around the development of R software packages for analyzing RNA sequencing (RNA-seq) and ChIP sequencing (ChIP-seq) high throughput genomic data. The first chapter describes the derfinder Bioconductor package which implements the DER Finder approach for identifying differentially expressed regions with RNA-seq data in an annotation-agnostic manner. The second chapter shows how derfinder can be applied to ChIP-seq data to identify differentially bounded regions. The third chapter describes the regionReport Bioconductor package for producing HTML or PDF reports from region-based genomic analyses, such as the derfinder analyses described in the previous chapters. The last thesis project Leonardo Collado-Torres carried out was the development of the recount2 resource and accompanying recount Bioconductor package using the Rail-RNA results from processing all the public human RNA-seq data at the time, which spans over 70,000 samples.

Affiliation

Lieber Institute for Brain Development, Johns Hopkins Medical Campus, Baltimore, Maryland, 21205, USA.

Education

2011-2016. Ph.D. in Biostatistics at Johns Hopkins Bloomberg School of Public Health, Baltimore, US. Advised by Jeff Leek and Andrew Jaffe.
2005-2009. Bachelor in Genomic Sciences (LCG) at the National Autonomous University of Mexico (UNAM), Cuernavaca, MX.
2002-2005. High school at ITESM Campus Cuernavaca, Cuernavaca, MX.

Academic and professional awards

2011: Awarded CONACyT Mexico scholarship for PhD studies outside Mexico.
2009: Summa Cum Laude for bachelor in Genomic Sciences studies at LCG-UNAM.
2005: Best high school average ($\sim$ 200 students): awarded ITESM system 90% scholarship for college studies, declined to join LCG-UNAM.

Professional experience

2016-current. Data Scientist with Andrew Jaffe lab at LIBD, Baltimore. At LIBD, Dr. Collado-Torres is part of the Data Science team which goals include better understanding and characterizing genomics signatures in the human brain, including DNA methylation and gene expression. Leonardo helps mentor other team members, provides support for LIBD projects and is advancing his academic career as part of Andrew Jaffe’s lab.
2009-2011. Bioinformatician with Enrique Morett lab at IBT-UNAM, Cuernavaca, MX. Identified transcriptions start sites and transcription units in Escherichia coli and Geobacter sulfurreducens with RNA-seq data. Developed the BacterialTranscription R package.
2009-2011. Scientific executive at Winter Genomics in Cuernavaca, MX. Responsible for recruiting and hiring new personnel, overseeing and supervising bioinformaticians, training new employees, writing research reports and presenting them to colleagues, and organizing all scientific projects.
2007-2009. Undergraduate research assistant at Guillermo Davila’s lab at CCG-UNAM, Cuernavaca, MX. Determined bacteriophage ecological groups by developing a method based on codon distribution of all phage sequenced genomes. Joint work with Sur Herrera Paredes.
2006-2007. Undergraduate research assistant at Roberto Kolter’s lab at Harvard, Boston, US. Supervisor: Elizabeth Shank. Carried out screenings to identify bacteria that activate the production of exopolysaccharide through the activation of the gene tasA in Bacillus subtilis.

References

Anders, Simon, and Wolfgang Huber. 2010. “Differential Expression Analysis for Sequence Count Data.” Genome Biology 11 (10): R106. https://doi.org/10.1186/gb-2010-11-10-r106.

Anders, Simon, Paul Theodor Pyl, and Wolfgang Huber. 2014. “HTSeq–a Python Framework to Work with High-Throughput Sequencing Data.” bioRxiv.

Collado-Torres, Leonardo, Abhinav Nellore, Alyssa C. Frazee, Christopher Wilks, Michael I. Love, Ben Langmead, Rafael A. Irizarry, Jeffrey T. Leek, and Andrew E. Jaffe. 2017. “Flexible Expressed Region Analysis for RNA-Seq with Derfinder.” Nucleic Acids Research 45 (2): e9–9. https://doi.org/10.1093/nar/gkw852.

Collado-Torres, Leonardo, Abhinav Nellore, Kai Kammers, Shannon E. Ellis, Margaret A. Taub, Kasper D. Hansen, Andrew E. Jaffe, Ben Langmead, and Jeffrey T. Leek. 2017. “Reproducible RNA-Seq Analysis Using Recount2.” Nature Biotechnology 35 (4): 319–21. https://doi.org/10.1038/nbt.3838.

Ellis, Shannon E, Leonardo Collado-Torres, and Jeffrey Leek. 2017. “Improving the Value of Public RNA-Seq Expression Data by Phenotype Prediction.” bioRxiv. https://doi.org/10.1101/145656.

Frazee, Alyssa C., Ben Langmead, and Jeffrey T. Leek. 2011. “ReCount: A Multi-Experiment Resource of Analysis-Ready RNA-Seq Gene Count Datasets.” BMC Bioinformatics 12: 449. https://doi.org/10.1186/1471-2105-12-449.

Frazee, Alyssa C, Sarven Sabunciyan, Kasper D Hansen, Rafael A Irizarry, and Jeffrey T Leek. 2014. “Differential Expression Analysis of RNA-seq Data at Single-Base Resolution.” Biostatistics (Oxford, England), January. https://doi.org/10.1093/biostatistics/kxt053.

Huber, Wolfgang, Vincent J Carey, Robert Gentleman, Simon Anders, Marc Carlson, Benilton S Carvalho, Hector Corrada Bravo, et al. 2015. “Orchestrating High-Throughput Genomic Analysis with Bioconductor.” Nature Methods 12 (2): 115–21. https://doi.org/10.1038/nmeth.3252.

Jaffe, Andrew E., Jooheon Shin, Leonardo Collado-Torres, Jeffrey T. Leek, Ran Tao, Chao Li, Yuan Gao, et al. 2015. “Developmental Regulation of Human Cortex Transcription and Its Clinical Relevance at Single Base Resolution.” Nature Neuroscience 18 (1): 154–61. https://doi.org/10.1038/nn.3898.

Jaffe, Andrew E., Richard Straub, Joo Heon Shin, Ran Tao, Yuan Gao, Leonardo Collado-Torres, Tony Kam-Thong, et al. 2017. “Developmental and genetic regulation of the human cortex transcriptome in schizophrenia.” bioRxiv.

Kim, Daehwan, Ben Langmead, and Steven L. Salzberg. 2015. “HISAT: A Fast Spliced Aligner with Low Memory Requirements.” Nature Methods 12 (4): 357–60. https://doi.org/10.1038/nmeth.3317.

Kim, Daehwan, Geo Pertea, Cole Trapnell, Harold Pimentel, Ryan Kelley, and Steven L Salzberg. 2013. “TopHat2: Accurate Alignment of Transcriptomes in the Presence of Insertions, Deletions and Gene Fusions.” Genome Biology 14 (4): R36. https://doi.org/10.1186/gb-2013-14-4-r36.

Law, Charity W, Monther Alhamdoosh, Shian Su, Gordon K Smyth, and Matthew E Ritchie. 2016. “RNA-seq analysis is easy as 1-2-3 with limma, Glimma and edgeR.” F1000Research 5 (May): 1408. https://doi.org/10.12688/f1000research.9005.2.

Love, Michael I., Simon Anders, Vladislav Kim, and Wolfgang Huber. 2016. “RNA-Seq workflow: gene-level exploratory analysis and differential expression.” F1000Research 4 (May): 1070. https://doi.org/10.12688/f1000research.7035.2.

Love, Michael I, Wolfgang Huber, and Simon Anders. 2014. “Moderated Estimation of Fold Change and Dispersion for RNA-Seq Data with DESeq2.” Genome Biology 15 (12): 1–21.

Nellore, Abhinav, Leonardo Collado-Torres, Andrew E. Jaffe, José Alquicira-Hernández, Christopher Wilks, Jacob Pritt, James Morton, Jeffrey T. Leek, and Ben Langmead. 2016. “Rail-RNA: Scalable Analysis of RNA-Seq Splicing and Coverage.” Bioinformatics (Oxford, England), September. https://doi.org/10.1093/bioinformatics/btw575.

Nellore, Abhinav, Andrew E. Jaffe, Jean-Philippe Fortin, José Alquicira-Hernández, Leonardo Collado-Torres, Siruo Wang, Robert A. Phillips Iii, et al. 2016. “Human Splicing Diversity and the Extent of Unannotated Splice Junctions Across Human RNA-Seq Samples on the Sequence Read Archive.” Genome Biology 17 (1): 266.

Robinson, Mark D, Davis J McCarthy, and Gordon K Smyth. 2010. “edgeR: A Bioconductor Package for Differential Expression Analysis of Digital Gene Expression Data.” Bioinformatics (Oxford, England) 26 (1): 139–40. https://doi.org/10.1093/bioinformatics/btp616.

Wilks, Christopher, Phani Gaddipati, Abhinav Nellore, and Benjamin Langmead. 2017. “Snaptron: querying and visualizing splicing across tens of thousands of RNA-seq samples.” bioRxiv. https://doi.org/10.1101/097881.

Y, Liao, Smyth GK, and Shi W. 2013. “The Subread Aligner: Fast, Accurate and Scalable Read Mapping by Seed-and-Vote.” Nucleic Acids Research 41: e108.

Using Slack for Academic Departmental Communication

Tue, 19 Jun 2018 00:00:00 +0000

This is a joint blog post between Stephanie Hicks and Leonardo Collado-Torres. We want to share with you our experience using Slack and why you should join us. This post is in an interview style.

{width=400px}

What is Slack?

[SH] Slack is a communication tool for teams. The main idea is you have individual chat rooms (referred to as channels that always begin with the # symbol), which are organized by topics. Traditionally, if an email is sent to everyone on your team, each person must decide how tag or organize emails in their account. In contrast, Slack provides the structure and organization for all users on the team. Each user clicks on a specific topic, or channel, and all the messages related to that topic are already there, thereby reducing the organizational burden. In a slack team used by academic department, topics can be anything from announcements of #conferences, #workshops, #jobs, #seminars, #working_groups, #good_reads, #food_callouts, #payroll, and #IT_help. You can write public messages in the individual channels or send private direct messages to anyone on your team. Everything in Slack is searchable (e.g. files, conversations, people). Finally, Slack includes a lot of functionality and integration with other useful tools such as Google Drive.

![](http://lcolladotor.github.io/post/2018-06-19-using-slack-for-academic-departmental-communication_files/Screen Shot 2018-06-19 at 4.00.33 PM.png){width=600px}

[LCT] Slack is a website that provides access to a group of message channels bundled together in what they call a workspace. These channels can have specific purposes and different sets of members. Slack also provides many integrations with other software that we commonly use, like integration with GitHub, todo lists, Google Calendar, Google Docs, etc. The idea of using Slack is to keep your communications with close colleagues organized and make them more fluid than email. You might use the general channel for making broad announcements, such as an interesting talk from a guest. Then switch to a private channel with your 4 colleagues for that new secret project you are working on and discuss some ideas that would be useful to explore in more detail. And finally, switch to a one-on-one channel with your advisor and ask if you can meet for a few minutes that later today. Like many websites, Slack is also available as its own desktop and phone applications.

How do you use Slack?

[SH] While I communicate with colleagues everyday on Slack, I think it’s important to distinguish between Slack and email. Both are important, I just use them for different purposes. I view communication on Slack to be for instant, real-time messaging and discussion to increase collaboration. These are usually short messages compassing a single idea. In contrast, I view email to be a slower, more deliberate medium. There is less expectation for immediate response. In both cases, you can always turn them off completely and enjoy some peace too!

![](http://lcolladotor.github.io/post/2018-06-19-using-slack-for-academic-departmental-communication_files/Screen Shot 2018-06-19 at 4.04.08 PM.png){width=600px}

versus

![](http://lcolladotor.github.io/post/2018-06-19-using-slack-for-academic-departmental-communication_files/Screen Shot 2018-06-19 at 4.04.18 PM.png){width=600px}

[LCT] I’m part of multiple Slack workplaces (collections of channels and users) but I mostly use one with my genomics collaborators. There, we have a channel per project, some of which can be silent for weeks/months. But that’s ok, because I can always jump back to them and revise what we were talking about. We have a channel for R questions, one for the team that I’m a part of (Andrew Jaffe’s lab), one for organizing when and where to eat lunch, another one with the recount2 team, a diversity channel, one for our computing cluster JHPCE, among others. I also frequently ask and get asked questions on one-on-one channels with Andrew’s lab members and people elsewhere from our genomics workplace. I love using Twitter for networking in academia and keeping myself updated. So I also tend to share relevant news to specific channels I’m a part of. For the project specific channels I also use the integration with GitHub to keep everything in one place and reduce the chance of git merge issues :P

What do you like about Slack?

[SH] I recently transitioned from being a postdoctoral research fellow to an assistant professor. Many people warned me about the increase in the number of emails that I would receive. While that is certainly true, I would attribute Slack to being the biggest mitigator in reducing the number of emails that I have received. It reduces my email burden and increases communication between collaborators and colleagues. When an email sometimes seems too formal, Slack is perfect for sending out quick messages.

[LCT] It greatly reduces the amount of emails and email chains. Also, I like having the option of more communication with all my colleagues than would be typically allowed via email. Like no one would like an email about some interesting R package tweet or emails about lunch. Also, it was very useful this past year when I advised two students. I could get them involved in different projects, get them observe how we do some things, but also provide them a space to ask me questions during the week outside of our weekly meeting slot. Ultimately, I like to be organized and Slack helps me stay organized.

Any more tips?

[SH] I have found that the more I engage and write messages in channels on a Slack team, the more I get in return. As conversations flow in a given channel on a particular topic, I may or may not have anything useful to contribute. If it’s the former, I try to pass along links, advice, knowledge or whatever is appropriate. If it’s the latter, I still gain knowledge by being able to read other people’s responses and learning how others would have approached or solved a problem. Either way it’s a win-win!

[LCT] Regardless of what channel you use, it’s important to keep your messages polite. Basically, following a code of conduct similar to this one. This is not the anonymous internet, here you are interacting with your colleagues and you have to be aware of biases you might have, otherwise you might perpetrate some problems like sexism. Remember, you share the responsibility of making sure that your Slack workspace is a welcoming environment where anyone is allowed to ask questions. I also think that it’s very important to keep your boundaries clear between work and home. I highly recommend going to your preferences and automatically disabling notifications outside work hours as shown below. That’s something you can’t do with Google Chat: yes, I might be available after work hours, but I want to choose whether to respond or not and I don’t want others to expect an immediate answer if they see me online.

![](http://lcolladotor.github.io/post/2018-06-19-using-slack-for-academic-departmental-communication_files/Screen Shot 2018-06-18 at 9.52.35 AM.png){width=600px}

If you want more, check this great presentation about Slack by Stephanie Hicks.

Acknowledgments

This blog post was made possible thanks to:

References

[1] H. Wickham, J. Hester, W. Chang, and J. Bryan. devtools: Tools to Make Developing R Packages Easier. R package version 2.4.5. 2022. URL: https://CRAN.R-project.org/package=devtools.

[2] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  callr           3.7.3      2022-11-02 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  crayon          1.5.2      2022-09-29 [1] CRAN (R 4.3.0)
##  devtools      * 2.4.5      2022-10-11 [1] CRAN (R 4.3.0)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  ellipsis        0.3.2      2021-04-29 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  fs              1.6.2      2023-04-25 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  htmlwidgets     1.6.2      2023-03-17 [1] CRAN (R 4.3.0)
##  httpuv          1.6.11     2023-05-11 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  later           1.3.1      2023-05-02 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  memoise         2.0.1      2021-11-26 [1] CRAN (R 4.3.0)
##  mime            0.12       2021-09-28 [1] CRAN (R 4.3.0)
##  miniUI          0.1.1.1    2018-05-18 [1] CRAN (R 4.3.0)
##  pkgbuild        1.4.2      2023-06-26 [1] CRAN (R 4.3.0)
##  pkgload         1.3.2      2022-11-16 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  prettyunits     1.1.1      2020-01-24 [1] CRAN (R 4.3.0)
##  processx        3.8.2      2023-06-30 [1] CRAN (R 4.3.0)
##  profvis         0.3.8      2023-05-02 [1] CRAN (R 4.3.0)
##  promises        1.2.0.1    2021-02-11 [1] CRAN (R 4.3.0)
##  ps              1.7.5      2023-04-18 [1] CRAN (R 4.3.0)
##  purrr           1.0.1      2023-01-10 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  remotes         2.4.2      2021-11-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo     1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  shiny           1.7.4      2022-12-15 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  urlchecker      1.0.1      2021-11-30 [1] CRAN (R 4.3.0)
##  usethis       * 2.2.1      2023-06-23 [1] CRAN (R 4.3.0)
##  vctrs           0.6.3      2023-06-14 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  xtable          1.8-4      2019-04-21 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Mindfulness

Mon, 11 Jun 2018 00:00:00 +0000

I recently participated for the first time in a silent retreat (6 hrs) as part of a Mindfulness-Based Stress Reduction course. I’ve really been enjoying this course and the experience of learning new ways (for me) to live better and enjoy life more. If you haven’t heard of mindfulness before (like me a few months ago), Wikipedia defines it as:

Mindfulness is the psychological process of bringing one's attention to
experiences occurring in the present moment, which can be developed
through the practice of meditation and other training.

In academia and science, but really anywhere, we can encounter many sources of stress (some examples below). We all deal with it in different ways and this MBSR course is just one that I’m liking right now and recommend. If there’s no course near you, check out the Full Catastrophe Living book that my course is based on¹.

All these tweets are threads, so you’ll have to open them to see them: click on the blue bird on the right side of each tweet.

I won an NSF grant yesterday, with three weeks to go before applying to tenure.

A thread👇
— Bryony DuPont (@BryonyDuPont) June 10, 2018

Just another barrier that us scientists in #developing #countries need to find ways to overcome. Not only do we have less funding, but also generating data that for other countries is routine is more expensive. #latin #american #scientists #plight https://t.co/Hln7N4x4iM
— Daniela Robles @daniela-oaks.bsky.social (@daniela_oaks) June 7, 2018

Thanks for sharing this Ted! @tladeras As with anything, but particularly with mental health, I think it’s important to know you are not alone and that some things we feel have names: impostor syndrome, anxiety, depression, etc. Our minds are very powerful! #MentalHealthMatters https://t.co/X1UIAznV8f
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) June 6, 2018

One of the exercises on our silent retreat was mindful eating. That is, eating slowly and focusing on our senses while eating food. This involved smelling the food, seeing all the colors it has, feeling all the different textures, and focusing on its taste. Plus, not talking to anyone. I had bought a lunch before the retreat (so I wouldn’t have to break the silence during it) that I normally don’t find interesting: some fruit and a chicken curry wrap from CVS. But wow, it was amazing that day! I greatly enjoyed how it tasted and all the different sensations I felt. I couldn’t believe it was the same food I had eaten other times before without much excitement.

Lately I’ve been processing some strong feelings related to feeling unwelcome, homesickness and loneliness. I’m doing many things that I know help me, starting with talking about them with my family. I’ve also been keeping myself busy doing some sports outside, going to events, bars, having friends over, etc. But what motivated me to write this post today is that after lunch I left really bad today. I had to go outside and try to calm myself.

I thought I did, returned to work, and nah, I couldn’t do it today. So I went home, rested in my room in the dark, and I find myself finishing this post about 2 hours later. I’m not sure it’s related to the MSBR course, but I’m wondering if the mindfulness practice is also opening my mind’s ears and eyes to my inner feelings way more than normal². I don’t think it’s bad. It’s just that it’s way harder to enjoy this process than it was eating a lunch. I’m trying to feel happy because I have feelings and I’m recognizing them. My family is also happy because they view it as a part of maturing and that I’m getting ready for taking another step in my life.

Just some things are taking me way more time than normal. For example, it took me 2 days to process what I wanted to reply to a tweet and how to word it: it turned out well because I think we all heard each other ^^.

Thanks for the link @apreshill! I agree that the blog post was aweful! You can’t blame X for the fact that X is marginalized. However, I disagree with the notion that Z can’t talk about X (challenging). I’m in favor of inclusion. Otherwise that line of thought leads to xenophobia
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) June 11, 2018

I also discovered that the silent retreat reminded me of vacations where I turn off my phone/email and spend a few days thinking about what’s next. I used to go to Mexico every 6 months or so during my PhD to recharge. I want to do it again, but I can’t. That lead me to my strong feelings of homesickness I had today.

Just to close, I’m OK. I appreciate all the support I have and I recommend others to learn about mindfulness. You might be missing out some super tasty lunches and/or an exploration trip towards your inner self.

I also want to add that I know some of you might see this post as over-sharing (yes, it might be). One reply is that I think that leaders also have to show their vulnerable sides. This has been debated a lot in academic twitter accounts in the context of showing your failures too, not just your successes.

Acknowledgments

This blog post was made possible thanks to:

References

\[1\] H. Wickham, J. Hester, W. Chang, and J. Bryan. devtools: Tools to Make Developing R Packages Easier. R package version 2.4.5. 2022. URL: https://CRAN.R-project.org/package=devtools.

\[2\] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  callr           3.7.3      2022-11-02 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  crayon          1.5.2      2022-09-29 [1] CRAN (R 4.3.0)
##  devtools      * 2.4.5      2022-10-11 [1] CRAN (R 4.3.0)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  ellipsis        0.3.2      2021-04-29 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  fs              1.6.2      2023-04-25 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  htmlwidgets     1.6.2      2023-03-17 [1] CRAN (R 4.3.0)
##  httpuv          1.6.11     2023-05-11 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  later           1.3.1      2023-05-02 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  memoise         2.0.1      2021-11-26 [1] CRAN (R 4.3.0)
##  mime            0.12       2021-09-28 [1] CRAN (R 4.3.0)
##  miniUI          0.1.1.1    2018-05-18 [1] CRAN (R 4.3.0)
##  pkgbuild        1.4.2      2023-06-26 [1] CRAN (R 4.3.0)
##  pkgload         1.3.2      2022-11-16 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  prettyunits     1.1.1      2020-01-24 [1] CRAN (R 4.3.0)
##  processx        3.8.2      2023-06-30 [1] CRAN (R 4.3.0)
##  profvis         0.3.8      2023-05-02 [1] CRAN (R 4.3.0)
##  promises        1.2.0.1    2021-02-11 [1] CRAN (R 4.3.0)
##  ps              1.7.5      2023-04-18 [1] CRAN (R 4.3.0)
##  purrr           1.0.1      2023-01-10 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  remotes         2.4.2      2021-11-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo     1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  shiny           1.7.4      2022-12-15 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  urlchecker      1.0.1      2021-11-30 [1] CRAN (R 4.3.0)
##  usethis       * 2.2.1      2023-06-23 [1] CRAN (R 4.3.0)
##  vctrs           0.6.3      2023-06-14 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  xtable          1.8-4      2019-04-21 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Baltimore friends, I have a paper copy if you want to borrow it for a while. ↩︎
It could be a coincidence with other events in my life. ↩︎

Getting ready to attend rOpenSci unconf18 and probably working on tidyverse-like functions for the first time

Fri, 27 Apr 2018 00:00:00 +0000

It’s Friday 7pm and it’s been a long week with ups and downs¹. But I’m enthused as I write this blog post. In less than a month from now I’ll be attending rOpenSci unconf18 and it’ll be my first time at this type of event. Yay!

Building on my streak of good news, I'm delighted to have been selected to attend @rOpenSci #Unconf18 https://t.co/Xe6lojB7TS ^_^ Also, thanks to the https://t.co/o5OwUWEaBD and @LieberInstitute for their support! I'm hoping to relay as much as I can to @LIBDrstats #rstats
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) April 11, 2018

In my self introduction to everyone attending, I mentioned that I don’t use the pipe (%>%) symbol and that I use <- for assignment.

Recently I had my pre-unconf chat with [Stefanie Butland](https://ropensci.org/about/#team) (read more about these chats [in her great blog post](https://blog.trelliscience.com/the-value-of-welcome-part-2-how-to-prepare-40-new-community-members-for-an-unconference/)). In my notes to Stefanie before our chat I had mentioned again my lack of R piping experience and we talked about it. As we talked, it became clear that a blog post on related topics would be useful. Sure, I could have asked directly to the other unconf18 participants, but maybe others from the R community in general can chime in or benefit from reading the answers.

Coding style and git

I’m attending unconf18 with an open mind and I think of myself as someone who can be quite flexible (not with my body!) and accommodating. I’m assuming that most participants at unconf18 will use = for assignment. I’m not looking to start any discussions about the different assignment operators. Simply, I am willing to use whatever the majority uses. Just like I did in my first pull request to the blogdown package (issue 263). I was trying to follow Yihui Xie’s coding style to make his life easier and have a clean (or cleaner) git history. From Yihui’s post on this pull request I can see that he liked it.

Keeping this in mind, I think that following the coding style of others will be something I’ll do at unconf18. I haven’t really worked in any R packages with many developers actively working on the package. But I imagine that setting a common coding style will minimize git conflicts, and no one wants those². I don’t know if we’ll all follow some common guidelines at unconf18. I actually imagine that it’ll be project-specific. Why? Well because you can create an R project in RStudio³ and set some defaults for the project such as:

the number of spaces for tab
line ending conversions

We can also set some global RStudio preferences like whether to auto-indent code after paste, length of lines.

Additionally, we can decide whether we’ll use the RStudio “wand” to *reformat code*.

Maybe all of this is unnecessary, maybe everyone will work in different non-overlapping functions and thus avoid git conflicts. For example, at my job sometimes I write code with = users, but we don’t work on the same lines of the code file. Later on it becomes easy to identify who wrote which line without having to use git blame (awful name, right?).

Tidyverse-like functions

So far, I think that these coding style issues are minor and will be easily dealt with. I think that we’ll all be able to adapt and instead focus on other problems (like whatever the package is trying to solve) and enjoy the experience (network, build trust as Stefanie put it).

My second concern has to do with something I imagine could require more effort: my homework before the unconf. That is, writing tidyverse-like functions. Like I’ve said, I haven’t used the R pipe %>% symbol. I’ve executed some code with it before, seeing it in many blog posts, but I’ve never actually written functions designed to be compatible with this type of logic.

If I help write a function that is not pipe-friendly, then it might not integrate nicely with other functions written by the rest of the team. It would lead to workarounds in the vignette or maybe having someone else re-factor my first function to make it pipe-friendly. Sure, I would learn from observing others make changes to my code. But I want to take advantage as much as I can from my experience at rOpenSci unconf⁴!

Since I don’t really use %>%, I’m unfamiliar with many things related to pipe-friendly (tidyverse-like) functions. For example:

Do you document them differently? Like make a single Rd file for multiple functions. Or do you make an Rd file per function even if the example usage doesn’t involve %>%?
I know that the first argument is important in pipe-friendly functions. But I ignore if the second and other arguments play a role or not.
Do people use the ellipsis (...) argument in pipe-friendly functions? With my derfinder package I ended up a very deep rabbit hole using .... I explained some of the logic in my dots blog post (there are fair criticisms to going deep with ... in the comments).
How do you write unit tests for pipe-friendly functions? Similar to how you write documentation for them, do the unit tests just test one function a time or do they test several at a time (that is the output after using %>%)?

These and other questions could involve time getting familiar with. Time that I could spend now, before unconf18, learning and at least getting a better sense of what to expect. Maybe I’m complicating myself and worrying too much about this. I imagine that the solution will involve a combination of:

Checking some popular tidyverse packages that use %>%. Like the vignettes/README files and examples.
Reading more about this in a book(s): I don’t know which one though.
Playing around a bit as a user with some of these functions. See what error messages pop up: actually I don’t know how users debug a series of functions tied together via %>%.

Wrapping up

I’m not saying everyone should learn about these topics before unconf18. I think that we are all (well, maybe excluding some) worried about not knowing $X$ or $Y$ R/git/travis/testthat/usethis/etc topic before unconf18. And that will part of why it’ll be great to meet everyone in what is known to be an extremely welcoming R conference ^^ (seriously, check all the unconf17 posts!).

## And I'm done proofreading the post. Yay!
Sys.time()

## [1] "2023-07-11 16:54:21 EDT"

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
devtools (Wickham, Hester, Chang, and Bryan, 2022)
knitcitations (Boettiger, 2021)

References

\[1\] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

\[3\] H. Wickham, J. Hester, W. Chang, and J. Bryan. devtools: Tools to Make Developing R Packages Easier. R package version 2.4.5. 2022. URL: https://CRAN.R-project.org/package=devtools.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  callr           3.7.3      2022-11-02 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  crayon          1.5.2      2022-09-29 [1] CRAN (R 4.3.0)
##  devtools      * 2.4.5      2022-10-11 [1] CRAN (R 4.3.0)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  ellipsis        0.3.2      2021-04-29 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  fs              1.6.2      2023-04-25 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  htmlwidgets     1.6.2      2023-03-17 [1] CRAN (R 4.3.0)
##  httpuv          1.6.11     2023-05-11 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  later           1.3.1      2023-05-02 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  memoise         2.0.1      2021-11-26 [1] CRAN (R 4.3.0)
##  mime            0.12       2021-09-28 [1] CRAN (R 4.3.0)
##  miniUI          0.1.1.1    2018-05-18 [1] CRAN (R 4.3.0)
##  pkgbuild        1.4.2      2023-06-26 [1] CRAN (R 4.3.0)
##  pkgload         1.3.2      2022-11-16 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  prettyunits     1.1.1      2020-01-24 [1] CRAN (R 4.3.0)
##  processx        3.8.2      2023-06-30 [1] CRAN (R 4.3.0)
##  profvis         0.3.8      2023-05-02 [1] CRAN (R 4.3.0)
##  promises        1.2.0.1    2021-02-11 [1] CRAN (R 4.3.0)
##  ps              1.7.5      2023-04-18 [1] CRAN (R 4.3.0)
##  purrr           1.0.1      2023-01-10 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  remotes         2.4.2      2021-11-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo     1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  shiny           1.7.4      2022-12-15 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  urlchecker      1.0.1      2021-11-30 [1] CRAN (R 4.3.0)
##  usethis       * 2.2.1      2023-06-23 [1] CRAN (R 4.3.0)
##  vctrs           0.6.3      2023-06-14 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  xtable          1.8-4      2019-04-21 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Feeling welcomed can be hard… oh well. ↩︎
We could talk about git for a long time too. But I hope that I’ll get by with some git push, git pull, and maybe git branch. ↩︎
It’s one of the sponsors http://unconf18.ropensci.org/#sponsors and well, probably want most will be using since it has such nice tools for writing R packages. ↩︎
Specially since most only attend one unconf, I think. So it’s different from other conferences that you can experience multiple years and network with the group across a longer period of time: that’s what I’ve done with the Bioconductor meetings. ↩︎

Latin American R/BioConductor Developers Workshop 2018

Thu, 19 Apr 2018 00:00:00 +0000

Today I’m excited to invite you to attend the Latin American R/BioConductor Developers Workshop 2018! It’ll be held in Cuernavaca, Mexico from July 30th to August 3rd, 2018. You can find the official announcement in the Bioconductor support website. Let me share with you why I’m excited about this workshop.

At BioC2017, Alejandro Reyes and I talked for a while about the low representation of Latin Americans through out the years that either of us have attended the BioC meetings¹. We wanted to see more Latin faces in person but also in the Bioconductor Support Website and among package contributors. We brainstormed for a while and a at dinner with Bioconductor core members we simply ran with our ideas: we brought up the subject and were pleasantly rewarded with support from the Bioconductor team (Alejandro did most of the convincing!). That is, if we organized a conference/workshop, they would help us by sending someone to be one of the instructors.

Previously, Alejandro and I had already discussed why we like the Bioc meetings so much. One of the keys is that the Bioc meetings have always had events for different levels of R users. That is, you can attend the conference each year and learn something new (that’s how I got my career started). So you can progress from learning R, to mastering several R packages, to contributing R packages to Bioconductor, to brainstorming future directions for the Bioconductor project². Additionally, the Bioc meetings have scientific talks in the mornings so that you can learn new scientific and R developments in a single conference. We’ve seen this model adapted in other Bioc events like BiocAsia, the European Bioconductor Meeting, etc. It would be ideal if we got to that point, but we wanted to start smaller with a focus on developing R packages.

Separately, Heladia Salgado, Romualdo Zayas and others from CCG-UNAM³ and NNB-UNAM⁴ have organized the “Talleres Internacionales de Bioinformática”⁵ (TIBs) for years now. Actually, I was a student in 2006 and gave a lecture in the summer 2010 event. At the TIBs they’ve had an introductory R class for a few years now where Selene Fernandez-Valverde and Alejandra Medina-Rivera have participated as instructors. Among the feedback they’ve received is the desire for a more advanced R course.

Alejandro and I talked to Alejandra Medina-Rivera who shared our enthusiasm for Bioconductor. As a PI in Mexico, Alejandra knows very well how sparse are the R/Bioconductor courses that she can send her students to. By the way, Alejandra is the one who first invited me to attend BioC2008!⁶ We quickly started an email thread with Heladia and basically, the pieces started to fall in the right places. Heladia, Delfino Garcia-Alonso, Daniela Ledezma, Laura Gómez, Shirley Alquicira, Thalia Uranga Martínez, among others have been instrumental in organizing the workshop. Heladia, Alejandra and others also secured crucial funding from CCG-UNAM and LIIGH-UNAM. Together we were able to invite and get all the logistics so that Martin Morgan, Heather Turner, Benilton Carvalho, Selene Fernandez-Valverde, María Teresa Ortiz and Alicia Mastretta Yanes could all help us teach part of the workshop and/or present their work.

Briefly, these are some of the reasons why they are amazing:

Martin: Bioconductor project leader.
Heather: extensive experience teaching R and bioinformatics.
Benilton: he helped organize a similar workshop in 2015: LAb Foundation Brasil. Also an academic relative of Alejandro and myself.
Selene: L’Oréal-UNESCO For Women in Science, has taught at TIBs before.
Teresa: RLadiesCDMX plus just look at her awesome GitHub profile!
Alicia: she has taught an intro to bioinformatics and reproducible genetic analyses course, plus check her GitHub profile!

If you check the workshop schedule we have, you’ll notice that we will have several sessions devoted to actually writing R packages. So the course will cover both the theory and the practice side of things. If you attend the workshop, you’ll have plenty of opportunities to network with your peers and to learn from other R developers. You should be super excited now!! ^^ Plus we have a code of conduct (en Español) to ensure that the conference will be welcoming to everyone.

This is also a reminder that you have to keep trying. Back in 2009 or 2010 I had gotten an offer of support from Bioc for sending someone to Cuernavaca, but due to funding circumstances it fell through. Like I said, all the pieces fell in the right places this time. Plus Heladia Salgado has made it all possible!

2018 is looking like a good year for R courses in Latin America. For example, there’s LatinR and Buenos Aires RLadies plus our workshop! Fingers crossed, I’ll be able to attend the workshop in 2019!

Again, I’m excited to invite you to attend the Latin American R/BioConductor Developers Workshop 2018! Regardless of whether you can attend it, help us spread the word:

Bioc Support website: https://support.bioconductor.org/p/108108/
Bioc twitter: https://twitter.com/Bioconductor/status/986943878535303168

#rstats / @Bioconductor Latin American BioC Developers Workshop 2018 https://t.co/eoL4O4V5xT
— Bioconductor (@Bioconductor) April 19, 2018

Tweet by Alejandro Reyes: https://twitter.com/areyesq/status/986943887976656896

If you are a programmer and want to become a @Bioconductor developer, join the Latin American Developers Workshop in Cuernavaca, Mexico, from Jul 30 to Aug 3. More info here: https://t.co/TAcfqYSiM6 #rstats @CDSBMexico @ccg_unam
— Alejandro Reyes (@areyesq) April 19, 2018

Tweet by LCG UNAM: https://twitter.com/lcgunam/status/986691090920230913

Talleres Internacionales de Bioinformática 2018 y Latin American R/Bioconductor Developers Workshop 30jul-3ago @ccg_unam @unammorelos @unam_mx https://t.co/oHWKjbrDrb #RegistroAbierto pic.twitter.com/bdmOREzhuy
— LCG UNAM (@lcgunam) April 18, 2018

CDSBMexico twitter: https://twitter.com/CDSBMexico/status/986672270964871169

Te invitamos a los Talleres Internacionales de Bioinformática 2018https://t.co/dsSekSp4U8 pic.twitter.com/UN1wmOF9SX
— ComunidadBioInfo (@CDSBMexico) April 18, 2018

CDSBMexico facebook: https://www.facebook.com/events/215830399185346/
Website: https://comunidadbioinfo.github.io/post/r-bioconductor-developers-workshop-2018/
Registration form: http://congresos.nnb.unam.mx/TIB2018/registro/ (select “Latin American R/BioConductor Developers Workshop 2018”)

The Bioc team has been great about financially supporting newcomers! I was one of them: just remember to apply before the deadline expires! ↩︎
It doesn’t have to be in that order. For example, anyone attending the Bioc meeting is welcome at the Developer’s Day and to chime in the discussions about the Bioconductor project. ↩︎
Centro de Ciencias Genómicas: Center for Genomic Sciences ↩︎
Nodo Nacional de Bioinformática: National Bioinformatics Node ↩︎
Internacional Bioinformatics Workshops ↩︎
I then taught a few courses at LCG-UNAM, CCG-UNAM and IBT-UNAM from 2008 to 2011 with some slides in Spanish and English. Alejandro was my TA for one of these courses! For a few years since then I would get emails in Spanish asking about some of those courses. You can still find those courses online, but I bet that the material is outdated. In any case, it’s another example of the limited training opportunities. ↩︎

blogdown archetype (template)

Thu, 08 Mar 2018 00:00:00 +0000

In a recent blog post I wrote about having a template for blogdown posts. I wanted to know if it was possible to do this and make my life (and others hopefully) easier for writing new blog posts that are ready to go with the features I frequently re-use.

In my case, I like using BiocStyle (Oleś, 2023) for functions such as CRANpkg(), Biocpkg() and Githubpkg(). I also like using knitcitations (Boettiger, 2021) for citing with citep() packages or papers; I use citation() and bib_metadata() to get the necessary information, respectively. Furthermore, I prefer devtools::session_info() (Wickham, Hester, Chang, and Bryan, 2022) over sessionInfo() since it provides information of where you got the package, which becomes specially relevant when using packages from GitHub. Finally, I like thanking the creators of the tools I use, which in this case is blogdown (Xie, Hill, and Thomas, 2017).

I also like reminding myself how to do some common tasks. Basically, the equivalent of the new R Markdown file you get when using RStudio. In my case, I want to remind myself of the YAML options I frequently use (toc, fig height and width) or how to add screenshots.

My first post on this topic is actually rather messy. That’s because at that time:

I didn’t know about hugo archetypes which are template files,
I hadn’t even thought of making the Insert Image addin.

I went down the rabbit hole of archetypes and blogdown, reported an issue resulated to this topic that was in the way of using archetypes for .Rmd posts. After some encouragement by Yihui Xie, I ended up fixing this issues in my first pull request ever to an RStudio package. The PR also added the Archetype option to the New Post RStudio addin (which I used right now ^^).

Creating my blogdown archetype (template)

To make my archetype (template) for blog posts I looked at the GitHub repo for the theme I’m using. It contains an archetypes directory with several files. I just looked at the one called post.md (check it here) and copied it from themes/hugo-academic/archetypes/post.md to archetypes/post.md. Next I added my favorite R code below the last +++ mark. You can check out the final version here. Below I display the version at the time of writing this blog post (I’m using a .txt extension otherwise the embedded gist doesn’t look good, but you want it to end in .md).

I couldn’t get the archetype to respect some of the YAML that blogdown adds, but well, that’s a single copy-paste I have to do now (if I actually decide to use the custom YAML options which are only for .Rmd posts).

I encourage you to make your own blogdown archetype (template). At least it should remind you to include reproducibility information which matters whenever any R code is involved.

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
devtools (Wickham, Hester, Chang, and Bryan, 2022)
knitcitations (Boettiger, 2021)

Yihui Xie also talked about my first PR in his blog.

References

[1] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

[3] H. Wickham, J. Hester, W. Chang, and J. Bryan. devtools: Tools to Make Developing R Packages Easier. R package version 2.4.5. 2022. URL: https://CRAN.R-project.org/package=devtools.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  callr           3.7.3      2022-11-02 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  crayon          1.5.2      2022-09-29 [1] CRAN (R 4.3.0)
##  devtools      * 2.4.5      2022-10-11 [1] CRAN (R 4.3.0)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  ellipsis        0.3.2      2021-04-29 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  fs              1.6.2      2023-04-25 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  htmlwidgets     1.6.2      2023-03-17 [1] CRAN (R 4.3.0)
##  httpuv          1.6.11     2023-05-11 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  later           1.3.1      2023-05-02 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  memoise         2.0.1      2021-11-26 [1] CRAN (R 4.3.0)
##  mime            0.12       2021-09-28 [1] CRAN (R 4.3.0)
##  miniUI          0.1.1.1    2018-05-18 [1] CRAN (R 4.3.0)
##  pkgbuild        1.4.2      2023-06-26 [1] CRAN (R 4.3.0)
##  pkgload         1.3.2      2022-11-16 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  prettyunits     1.1.1      2020-01-24 [1] CRAN (R 4.3.0)
##  processx        3.8.2      2023-06-30 [1] CRAN (R 4.3.0)
##  profvis         0.3.8      2023-05-02 [1] CRAN (R 4.3.0)
##  promises        1.2.0.1    2021-02-11 [1] CRAN (R 4.3.0)
##  ps              1.7.5      2023-04-18 [1] CRAN (R 4.3.0)
##  purrr           1.0.1      2023-01-10 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  remotes         2.4.2      2021-11-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo     1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  shiny           1.7.4      2022-12-15 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  urlchecker      1.0.1      2021-11-30 [1] CRAN (R 4.3.0)
##  usethis       * 2.2.1      2023-06-23 [1] CRAN (R 4.3.0)
##  vctrs           0.6.3      2023-06-14 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  xtable          1.8-4      2019-04-21 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

blogdown Insert Image addin

Wed, 07 Mar 2018 00:00:00 +0000

Have you ever tried inserting an image into a blogdown post? Maybe you have, or maybe you tried and gave up. Lets first review the hard way before getting to the solution I contributed.

The hard way

The process involves copying the target image to the static directory that corresponds to the blogdown post. Lets say that your post is called 2018-03-07-my-new-post.Rmd and lives at content/post/, so it’s full path is content/post/2018-03-07-my-new-post.Rmd. When you run the RStudio blogdown addin Serve Site, behind curtains the directory static/post/2018-03-07-my-new-post_files is created and inside it you can find the images made by your R code: likely at static/post/2018-03-07-my-new-post_files/figure_html.

So far everything is working! But now you want to add a screenshot or some other image to your blog post. Lets say that your image is ~/Desktop/screenshot.jpg. Your ~/Desktop directory is not part of your blogdown directory and well, simply put, your website won’t find the image. We need to put it in a location that will be made public by hugo. That is, we need to put it inside static/post/2018-03-07-my-new-post_files (or anywhere inside static, but we like to keep things tidy!).

Ok, so we copy our screenshot file ~/Desktop/screenshot.jpg and save it as static/post/2018-03-07-my-new-post_files/screenshot.jpg. The next time we render our site and publish it, the figure will be available in the web. But it’s still not part of our blogdown post.

So we need to use either the Markdown or HTML syntax for adding the image. Maybe your initial thought is to use:

![](screenshot.jpg)

Except that will not work. We need to use almost all the path (just remove static) as shown below:

![](/post/2018-03-07-my-new-post_files/screenshot.jpg)

If you want to edit the height or width, then you need to use the HTML syntax. Something like:

<img alt = 'my new screenshot' width='200' src='/post/2018-03-07-my-new-post_files/screenshot.jpg' />

hard way notes

You could have also used knitr::include_graphics() and let blogdown copy it to the final location in static and link to it appropriately. However, you would have to keep your original images organized in a way that won’t bother hugo.

Another option that I used for a while, even in the days when my blog was based on Jekyll, is to render the figures yourself and copy the directory with the figures, plus mess around with how they are linked from R. Details here. Not something I recommend doing now.

Insert Image addin: aka, the easy way

If you are using blogdown, you most likely (you should if you can) are using RStudio and the great blogdown addins: New Post and Serve Site. I just recently started using them in the past few days and looking at the code I realized that it should be possible to make an addin that lets you:

select a target image,
copies the target image to the appropriate location under static,
gives you the correct code for linking the image.

Yihui Xie loved the idea (I think it’s fair to say that ^^) and helped me polish it in the pull request that implements it. He then refined the code even more!

Features of the Insert Image addin

The final features, at least as implemented in blogdown version 0.5.7 are:

Select an image from anywhere in your computer.
Automatically generate a candidate final location for your image under static, which you can edit. Useful if you want to rename the final figure.
Allow specifying the alternate description of the image (alt), height and width.
If the target image file exists, a dynamic menu shows up that asks you whether to overwrite it or not.
The final syntax is Markdown unless a width or height are used, in which case it uses HTML code.

Yihui Xie hinted at other possible future features, which maybe you can help implement.

Using the Insert Image addin

Step 1: install appropriate blogdown version

First of all, at the time of writing this post, you need the development version of blogdown. You can install it with:

## Check if you have version 0.5.7 or newer
## I actually used version 0.5.9 for this blog post
packageVersion('blogdown')

## If not, then get it!
##### If necessary:
## install.packages('devtools')
devtools::install_github('rstudio/blogdown')

You also need an up to date version of RStudio and I recommend also using R 3.4.x (or newer if you are reading this in the future). Re-start RStudio so it loads the new version of blogdown.

Step 2: open the Insert Image addin

Second, go to the Addins menu in the top section of the RStudio window and select the Insert Image blogdown addin.

Step 3: choose figure and inputs

So far the Insert Image addin looks like this:

So lets go head and select an image we want to upload. In my case, I chose an image that already exists.

You can rename the figure if you want, and if it doesn’t exist, the overwrite option goes away.

Step 4: hit done!

Lets go ahead and click done! Our text window in RStudio will insert the appropriate code for linking the image. In this case, it’s the following code:

<img src="/post/2018-03-07-blogdown-insert-image-addin_files/screenshot.png" alt="final image" width="400"/>

and this is the image:

Optional step 5

Now use the Serve Site addin and check if you like your images. You might want to change the height/widths or alternate text. You could also wrap the HTML/Markdown code around it for linking to a website.

You can also delete your original images, if for example, they are cluttering your ~/Desktop.

Conclusions

I hope that you will find this new addin as useful as I’m finding it, or even more. Plus hopefully this blog post gives you an idea of the difficulties before this addin existed. Also, I want to thank Yihui Xie for guiding me, I’ve learnt quite a bit recently. Though I will still use <- assignment operator for my own code hehe.

Acknowledgments

This blog post was made possible thanks to:

BiocStyle (Oleś, 2023)
blogdown (Xie, Hill, and Thomas, 2017)
devtools (Wickham, Hester, Chang, and Bryan, 2022)
knitcitations (Boettiger, 2021)

as well as Yihui Xie’s help and encouragement in the form of a signed sticker and the Great Hacker title ^^. He also wrote a blog post about the Insert Image addin.

References

[1] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

[3] H. Wickham, J. Hester, W. Chang, and J. Bryan. devtools: Tools to Make Developing R Packages Easier. R package version 2.4.5. 2022. URL: https://CRAN.R-project.org/package=devtools.

[4] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  BiocManager     1.30.21    2023-06-10 [1] CRAN (R 4.3.0)
##  BiocStyle     * 2.28.0     2023-04-25 [1] Bioconductor
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  callr           3.7.3      2022-11-02 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  crayon          1.5.2      2022-09-29 [1] CRAN (R 4.3.0)
##  devtools      * 2.4.5      2022-10-11 [1] CRAN (R 4.3.0)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  ellipsis        0.3.2      2021-04-29 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  fs              1.6.2      2023-04-25 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  htmlwidgets     1.6.2      2023-03-17 [1] CRAN (R 4.3.0)
##  httpuv          1.6.11     2023-05-11 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  later           1.3.1      2023-05-02 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  memoise         2.0.1      2021-11-26 [1] CRAN (R 4.3.0)
##  mime            0.12       2021-09-28 [1] CRAN (R 4.3.0)
##  miniUI          0.1.1.1    2018-05-18 [1] CRAN (R 4.3.0)
##  pkgbuild        1.4.2      2023-06-26 [1] CRAN (R 4.3.0)
##  pkgload         1.3.2      2022-11-16 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  prettyunits     1.1.1      2020-01-24 [1] CRAN (R 4.3.0)
##  processx        3.8.2      2023-06-30 [1] CRAN (R 4.3.0)
##  profvis         0.3.8      2023-05-02 [1] CRAN (R 4.3.0)
##  promises        1.2.0.1    2021-02-11 [1] CRAN (R 4.3.0)
##  ps              1.7.5      2023-04-18 [1] CRAN (R 4.3.0)
##  purrr           1.0.1      2023-01-10 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  remotes         2.4.2      2021-11-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo     1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  shiny           1.7.4      2022-12-15 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  urlchecker      1.0.1      2021-11-30 [1] CRAN (R 4.3.0)
##  usethis       * 2.2.1      2023-06-23 [1] CRAN (R 4.3.0)
##  vctrs           0.6.3      2023-06-14 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  xtable          1.8-4      2019-04-21 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

R markdown blog template

Sat, 17 Feb 2018 00:00:00 +0000

This blog post is mostly for myself but maybe it’s useful to others. It contains my current R markdown blog template. I initially posted this as a question at StackOverflow. Then I read how much a burden we put in Yihui Xie and decided that my current setup (copy-pasting) works just fine. In any case using blogdown with the RStudio IDE is much simpler than what I used to do in the past with jekyll or with even my prior setup with blogdown.

Bibliography setup

First I define the citation information I’ll need. By the way, I used FAQ 7 for showing the R code chunk.

    ```{r bibsetup, echo=FALSE, message=FALSE, warning=FALSE}
    ## Load knitcitations with a clean bibliography
    library('knitcitations')
    cleanbib()
    cite_options(hyperlink = 'to.doc', citation_format = 'text', style = 'html')
    
    bib <- c('knitcitations' = citation('knitcitations'),
             'blogdown' = citation('blogdown')[2])
    ```

Post content

This is where I typically start to edit since the bibliography chunk is hidden.

R image

    ```{r 'plot'}
    ## This will create the /post/*_files/ directory
    ## where you can later copy the non-R images you want to use
    ## in the blog post
    plot(1:10, 10:1)
    ```

Note that I modified the YAML portion of the post to set the figure width and height. You can also include a table of contents if you want, though it affects the summary of the post. Check the output format section of the blogdown book for more details (Xie, Hill, and Thomas, 2017), including differences between .Rmd and .Rmarkdown files. Note that you can also use a _output.yml file as described there.

output:
  blogdown::html_page:
    toc: no
    fig_width: 5
    fig_height: 5

Custom image syntax

Here I remind myself of different ways I can include external images. Check blogdown issue 239 for some background information.

Markdown syntax for custom image:

![](/post/2018-02-17-r-markdown-blog-template_files/LIBD.jpg)

HTML syntax for centering image, including a link, and re-sizing the image to a fix width of 200 px.

<center>
<a href="http://lcolladotor.github.io/"><img alt = 'some website' width='200' src='/post/2018-02-17-r-markdown-blog-template_files/LIBD.jpg' /></a>
</center>

Reproducibility

## Reproducibility info
library('devtools')

## Loading required package: usethis

options(width = 120)
session_info()

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       macOS Ventura 13.4
##  system   aarch64, darwin20
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-07-11
##  pandoc   3.1.5 @ /opt/homebrew/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.3.0)
##  bibtex          0.5.1      2023-01-26 [1] CRAN (R 4.3.0)
##  blogdown        1.18       2023-06-19 [1] CRAN (R 4.3.0)
##  bookdown        0.34       2023-05-09 [1] CRAN (R 4.3.0)
##  bslib           0.5.0      2023-06-09 [1] CRAN (R 4.3.0)
##  cachem          1.0.8      2023-05-01 [1] CRAN (R 4.3.0)
##  callr           3.7.3      2022-11-02 [1] CRAN (R 4.3.0)
##  cli             3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
##  colorout        1.2-2      2023-05-06 [1] Github (jalvesaq/colorout@79931fd)
##  crayon          1.5.2      2022-09-29 [1] CRAN (R 4.3.0)
##  devtools      * 2.4.5      2022-10-11 [1] CRAN (R 4.3.0)
##  digest          0.6.31     2022-12-11 [1] CRAN (R 4.3.0)
##  ellipsis        0.3.2      2021-04-29 [1] CRAN (R 4.3.0)
##  evaluate        0.21       2023-05-05 [1] CRAN (R 4.3.0)
##  fastmap         1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
##  fs              1.6.2      2023-04-25 [1] CRAN (R 4.3.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.3.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
##  highr           0.10       2022-12-22 [1] CRAN (R 4.3.0)
##  htmltools       0.5.5      2023-03-23 [1] CRAN (R 4.3.0)
##  htmlwidgets     1.6.2      2023-03-17 [1] CRAN (R 4.3.0)
##  httpuv          1.6.11     2023-05-11 [1] CRAN (R 4.3.0)
##  httr            1.4.6      2023-05-08 [1] CRAN (R 4.3.0)
##  jquerylib       0.1.4      2021-04-26 [1] CRAN (R 4.3.0)
##  jsonlite        1.8.7      2023-06-29 [1] CRAN (R 4.3.0)
##  knitcitations * 1.0.12     2021-01-10 [1] CRAN (R 4.3.0)
##  knitr           1.43       2023-05-25 [1] CRAN (R 4.3.0)
##  later           1.3.1      2023-05-02 [1] CRAN (R 4.3.0)
##  lifecycle       1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
##  lubridate       1.9.2      2023-02-10 [1] CRAN (R 4.3.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
##  memoise         2.0.1      2021-11-26 [1] CRAN (R 4.3.0)
##  mime            0.12       2021-09-28 [1] CRAN (R 4.3.0)
##  miniUI          0.1.1.1    2018-05-18 [1] CRAN (R 4.3.0)
##  pkgbuild        1.4.2      2023-06-26 [1] CRAN (R 4.3.0)
##  pkgload         1.3.2      2022-11-16 [1] CRAN (R 4.3.0)
##  plyr            1.8.8      2022-11-11 [1] CRAN (R 4.3.0)
##  prettyunits     1.1.1      2020-01-24 [1] CRAN (R 4.3.0)
##  processx        3.8.2      2023-06-30 [1] CRAN (R 4.3.0)
##  profvis         0.3.8      2023-05-02 [1] CRAN (R 4.3.0)
##  promises        1.2.0.1    2021-02-11 [1] CRAN (R 4.3.0)
##  ps              1.7.5      2023-04-18 [1] CRAN (R 4.3.0)
##  purrr           1.0.1      2023-01-10 [1] CRAN (R 4.3.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
##  Rcpp            1.0.10     2023-01-22 [1] CRAN (R 4.3.0)
##  RefManageR      1.4.0      2022-09-30 [1] CRAN (R 4.3.0)
##  remotes         2.4.2      2021-11-30 [1] CRAN (R 4.3.0)
##  rlang           1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
##  rmarkdown       2.23       2023-07-01 [1] CRAN (R 4.3.0)
##  rstudioapi      0.14       2022-08-22 [1] CRAN (R 4.3.0)
##  sass            0.4.6.9000 2023-05-06 [1] Github (rstudio/sass@f248fe5)
##  sessioninfo     1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
##  shiny           1.7.4      2022-12-15 [1] CRAN (R 4.3.0)
##  stringi         1.7.12     2023-01-11 [1] CRAN (R 4.3.0)
##  stringr         1.5.0      2022-12-02 [1] CRAN (R 4.3.0)
##  timechange      0.2.0      2023-01-11 [1] CRAN (R 4.3.0)
##  urlchecker      1.0.1      2021-11-30 [1] CRAN (R 4.3.0)
##  usethis       * 2.2.1      2023-06-23 [1] CRAN (R 4.3.0)
##  vctrs           0.6.3      2023-06-14 [1] CRAN (R 4.3.0)
##  xfun            0.39       2023-04-20 [1] CRAN (R 4.3.0)
##  xml2            1.3.4      2023-04-27 [1] CRAN (R 4.3.0)
##  xtable          1.8-4      2019-04-21 [1] CRAN (R 4.3.0)
##  yaml            2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

References

Citations made with knitcitations (Boettiger, 2021) and blog built using blogdown (Xie, Hill, and Thomas, 2017).

## Chunk normaly with options:  results = 'asis', echo = FALSE, cache = FALSE
## Print bibliography
bibliography(style = 'html')

[1] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.12. 2021. URL: https://CRAN.R-project.org/package=knitcitations.

[2] Y. Xie, A. P. Hill, and A. Thomas. blogdown: Creating Websites with R Markdown. Boca Raton, Florida: Chapman and Hall/CRC, 2017. ISBN: 978-0815363729. URL: https://bookdown.org/yihui/blogdown/.

Updating R

Thu, 04 May 2017 00:00:00 +0000

As you might know by now, the latest R version was recently released (R 3.4.0). That means that you are highly encouraged to update your R installation. There are several ways to do this some of which are documented in these other blog posts: Tal Galili, 2013, Kris Eberwein, 2015. You would think that it’s just a matter of downloading the latest R installer for your OS, installing it, and continuing your analysis. The reality is a bit more complicated. The following short steps will make your life easier.

Save your list of currently installed packages

If you want to continue using R with all the packages you previously had installed, the best way is to save the list of packages you currently have before installing a new R version. You can do so with these lines of code:

## Change accordingly
list_dir <- '/Users/lcollado/Dropbox/Computing/R'

## Get the list of installed packages
installed <- dir(.libPaths())

## Save the list for later use
save(installed, file = file.path(list_dir, paste0(Sys.Date(), '-installed.Rdata')))

## Explore the list
head(installed)

## [1] "abind"   "acepack" "ada"     "AER"     "affy"    "affyio"

length(installed)

## [1] 611

Install latest R

Ok, now you have a list of installed packages. It’s like a restore point. Next, you need to get the latest R installer for your OS from CRAN and install the latest R. For example, for a Mac that would be R-3.4.0.pkg. Install it as usual.

Restore your packages

By now you have a new R version installed but without all your favorite packages. So, how do you install them? You just need to open your latest list of installed packages and install them. Now, I’m a Bioconductor user which means that some of my packages are not on CRAN. But the following code will work for you even if all the packages you use are from CRAN.

## Change accordingly
list_dir <- '/Users/lcollado/Dropbox/Computing/R'

## Find the corresponding Rdata files
previous <- dir(path = list_dir, pattern = 'installed.Rdata')

## Load the latest one
load(file.path(list_dir, previous[length(previous)]))

## Just checking it
head(installed)

## [1] "abind"   "acepack" "ada"     "AER"     "affy"    "affyio"

Next, get the list of current R packages you have installed. Every new R installation comes with a few of them (the base packages). You don’t need to install those.

current <- dir(.libPaths())

Finally, install the missing packages

## For Bioconductor and CRAN packages
install.packages("BiocManager")
BiocManager::install(installed[!installed %in% current])

and now you can continue on with your analysis 😄 You didn’t even need to figure out the best order to install the packages!

GitHub packages

Some of your favorite R packages might only exist via GitHub. This list is likely short since most packages get distributed via CRAN. But if that’s the case, you can see which packages are missing by running:

## Check which packages are missing
current_post_installation <- dir(.libPaths())
installed[!installed %in% current_post_installation]

For example, in my case I use the colorout package which lives only in GitHub. I have to install that one manually:

install.packages('devtools')
library('devtools')
install_github("jalvesaq/colorout")

Other times a package might not be compiling for the new R version or might no longer be supported (defunct).

Misc for Bioconductor developers

If you are a Bioconductor developer or are planning on becoming one, then you need 2 versions of R at all times. One R for the bioc-release branch and another one for the bioc-devel branch. Sometimes it’s the same R version sometimes it’s not depending on the month of the year. Right now, Bioc-release (3.5) uses R 3.4.0 and Bioc-devel (3.6) also uses R 3.4.0. R Switch for Mac users will be your friend. I can’t find the old bioc-devel mailing list thread where I first learned this, but the idea is to download the latest R tar ball, change the name from 3.4 to something else (3.4devel in my case), put it back together into a tar ball and then use this tar ball to install a second R version.

## Download latest R tarball
wget http://r.research.att.com/el-capitan/R-3.4-branch/R-3.4-branch-el-capitan-sa-x86_64.tar.gz

## Un-tar it
tar -xvf R-3.4-branch-el-capitan-sa-x86_64.tar.gz

## Renamed files from 3.4 to 3.4 devel
mv Library/Frameworks/R.framework/Versions/3.4 Library/Frameworks/R.framework/Versions/3.4devel

## Put it back in a tar ball
tar -cvzf Rlib.tgz Library

## Install it
sudo tar fvxz Rlib.tgz -C /

There you go:

Reproducibility

## Reproducibility info
library('devtools')
options(width = 120)
session_info()

## Session info -----------------------------------------------------------------------------------------------------------

##  setting  value                       
##  version  R version 3.4.0 (2017-04-21)
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  tz       America/New_York            
##  date     2017-05-04

## Packages ---------------------------------------------------------------------------------------------------------------

##  package       * version date       source        
##  BiocInstaller * 1.27.2  2017-05-04 Bioconductor  
##  devtools      * 1.12.0  2016-12-05 CRAN (R 3.4.0)
##  digest          0.6.12  2017-01-27 CRAN (R 3.4.0)
##  evaluate        0.10    2016-10-11 cran (@0.10)  
##  knitr         * 1.15.1  2016-11-22 cran (@1.15.1)
##  magrittr        1.5     2014-11-22 cran (@1.5)   
##  memoise         1.1.0   2017-04-21 CRAN (R 3.4.0)
##  stringi         1.1.5   2017-04-07 cran (@1.1.5) 
##  stringr         1.2.0   2017-02-18 cran (@1.2.0) 
##  withr           1.0.2   2016-06-20 CRAN (R 3.4.0)

Want more?

Check other @jhubiostat student and alumni blogs at Bmore Biostats as well as topics on #rstats.

How to ask for help for Bioconductor packages

Mon, 06 Mar 2017 00:00:00 +0000

tl;dr Please post your question at the Bioconductor support website https://support.bioconductor.org/ and check the posting guide http://www.bioconductor.org/help/support/posting-guide/. It’s important that you provide reproducible code and information about your R session.

Recently I have been getting more questions about several packages I maintain. It’s great to see more interest from users, but at the same time most questions lack the information I need to help the users. I have also gotten most of the questions via email, which is why I am writing this post. As of today, I will no longer answer questions related to my Bioconductor packages via personal emails. This might sound harsh, but hopefully the rest of this post will convince you that it’s the best thing to do. You might also be interested in the basics of using derfinder, regionReport or recount, among others.

The Bioconductor project is a community project and it benefits from users interacting in public venues. When a user asks a question at the Bioconductor support website, they are providing information that future users might be interested in. That is, the user (U1) is contributing information to the overall documentation around the Bioconductor package they are asking a question about. Ideally, a new user (U2) can then read through the question U1 wrote, check the solution, and move on. This is one of the main reasons why we (developers) want questions to be well documented. There are a couple of quick things that U1 can check that will make their question much more useful to the community.

Session information

One of the strengths of Bioconductor is that all the packages have vignettes and lots of documentation. The packages are also checked regularly and must pass some tests. That also means that packages can change frequently, at least more frequently than CRAN packages. There’s also the added complexity that at any given point in time there is a release branch and a development branch. This means that there are many variables and saying that you are using the “latest version” doesn’t mean much to the developer. All of this information and more is part of the R session information. That is why I and others request users to post their session information. It’s very easy to get, simply run the following code:

## Install devtools if needed
# install.packages('devtools')

## Reproducibility info
library('devtools')
options(width = 120)
session_info()

The output might be too long to post in the Bioconductor support website. The easy solution is to save the information you want displayed in a gist. Then simply add the gist link in your question. Note that you need to have the link under “text” formatting and not “code”.

Code to reproduce the error

If U1 includes the session information, their question will be pretty good, but not ideal yet. Many of the questions I’ve been asked do not include code for me to figure out the exact steps of what they were doing. A lot of times I can infer pieces of what they were doing from their description of the problem. But doing so takes quite a bit of my time and effort, and is still not perfect. Now imagine that U2 is reading through the question: they would probably get lost!

There is a wide range of things that U1 could have done. To help the developer, the best thing is for the user to include the code that lead to the error. The code should include how the data was loaded, so that the developer can run it themselves and check in more detail what went wrong. This means providing a small subset of the data or using some publicly available data.

I realize that writing code that reproduces the error is not easy. But it helps a lot for learning more about R and Bioconductor. I can tell you that I went through the same process, and in my experience you can find out what you are doing wrong by writing the reproducible code.

Extra

Here are some other tips that are useful.

If you run traceback() immediately after getting the error and include the output in your question, that would be great. It makes it easier to check at what point the code failed and produced the error.
Recently when I ask questions myself, I include the “non-evaluated code” (clean code in your script) and “evaluated code” (think of the R console: a mix of code and output). The non-evaluated code makes it easier for others to copy-paste the code into their R session without having to deal with any formatting issues (example).
If you encounter a new error, post a new question instead of “replying” to the first one.
Introduce yourself.
Be polite.

By now you should be ready to post some great questions! Thanks for contributing to the Bioconductor community.

Want more?

Check other @jhubiostat student and alumni blogs at Bmore Biostats as well as topics on #rstats.

An alternative to the Mexico-US wall where the US would gain millions of dollars

Wed, 25 Jan 2017 02:00:00 +0000

tl;dr There is a 600 million to 2 billion USD annual market related to crossing the Mexico-US border. Allow temporary work visas (say 3 years) to take over this market and use the money to boost the US Border Patrol to build a wall of eyes, not a physical wall.

President Trump of the United States of America,

cc President Peña Nieto of the United Mexican States

Today, Wednesday January 25th 2017, you are expected to announce your plans about building a wall between the United States and Mexico. I am opposed to building that wall but I also believe that providing alternatives is important when disagreeing. With that in mind, let me expose an alternative to your physical wall.

Mexico has a net immigration rate of -1.7 migrants per 1,000 and a population of 123,166,749 individuals¹ which means that about 209,000 people leave Mexico every year. The Department of Homeland Security 2014 report² shows that 350,177 out of 679,996 (51%) total aliens apprehended are from Mexico (52% reported elsewhere³,⁴). Many of the illegal immigrants pay people for helping them cross the Mexico-US border also known as coyotes. Some informal surveys put the cost of a coyote between 3,000 and 20,000 US dollars⁵,⁶. That means that there is an annual market worth about 627 to 1,050 million US dollars. This market exists and has been steady for years now. A physical wall in the Mexico-US border might not stop the illegal immigration to the US. Loss of jobs in Mexico directly related to your policies might even prompt more people to leave Mexico⁷. I believe that the high volume of people crossing the Mexico-US border makes it easier for drug cartels to hide and transport drugs. That is, it makes it harder for the US Border Patrol Agents to find who is transporting drugs and the track down the routes they use.

So, what if the US took over this multi million dollar market? I have had the opportunity to visit countries that offer temporary work visas (ranging from 6 months to 3 years). The people interested in these visas have to pay a fee, pass a background check and sometimes go through health screenings. I believe that these immigrants pay the fee because it allows them to legally migrate, work for a while, save some money, and then go back to their home countries. From an immigrant’s perspective, it must be very appealing to pay the same amount of money it cost to migrate safely and legally (for temporary work) than to attempt to cross the border illegally with the risk of dying in the process or getting kidnapped. Additionally, I think that temporary workers would feel much safer to go out of their homes while in the US, spending some of their hard earned money in the process and stimulating the local economy. They would be more likely to get a regular job where they would pay income taxes instead of getting payed under the table and dodging the IRS. Furthermore, having the opportunity to legally go back home and visit family would be a huge advantage from an immigrant’s perspective.

While writing this letter I realized that the idea of immigration tariffs is not new⁸,⁹,¹⁰,¹¹. However, I have not heard it being discussed recently and simply ask that you consider it.

To try to convince you that immigration tariffs are an interesting alternative, consider what your government could do with an additional 600 to 1,000 million US dollars a year – or multiply those numbers by two to get 1.2 to 2 billions USD a year by taking into account that about 50% of illegal immigrants are Mexicans. The average Border Patrol Agent self-reported salary is 80,250 USD a year¹². The actual salaries range from 21,616 to 157,257 USD as of January 2017¹³ depending on the grade and level as well as other options¹⁴. In 2016, there were nearly 20,000 border patrol agents in the US¹⁵. That means that you could decide to increase the number of Border Patrol Agents by 10,000 (50%) for about 803 million USD or increase the number of Agents by 5,000 while increasing their salaries by 10% for about 602 million USD. Why do I bring up the number of Agents? Because that would mean that you are building a wall of eyes in the Mexico-US border. I believe that under this scenario, you and your public relations team could convince your supporters that you delivered on your campaign promise of building a wall. It would just be a different kind of wall. The extra Border Patrol Agents and/or happier Agents (due to their increased salary) along with fewer attempts to cross the border (because immigrants would be interested in paying the immigration tariff) would make them more efficient in their jobs. Some of them could be dedicated to making sure no one overstays their temporary work visa. Also note that by law, you have to be a US citizen to become a Border Patrol Agent¹⁶, so you would be increasing US jobs by 5 to 10 thousand under this scenario. That is more than the 700 jobs you claim to have saved from moving to Mexico from the US by cancelling the plans to build a factory in Mexico¹⁷. I also imagine that this scenario would boost bus and airplane ticket sales.

I am of the opinion that immigration tariffs would allow many individuals to have hope that they can improve their economic situation for them and their families. It would also translate hope into a specific monetary goal. As of yesterday, 3,000 USD is about 64,000 MXN¹⁸ that translates to 800 days of minimum wage salary in Mexico¹⁹. Saving that amount of money would take over 2 years if the person didn’t spend any of their income! While ideally the immigration tariff would be lower than 3,000 USD, ultimately there would be a threshold that opens doors to more economic opportunity. So, Peña Nieto and Mexico have the challenge of showing that Mexicans can stay in Mexico and improve their economical situation if they can save that amount of money to start a small business, advance their education, among other options.

Migration involves two sides and as presidents of two countries of united states (American and Mexican, respectively) I encourage you to keep thinking how to unitedly address migration and take over the multi million dollar market that coyotes currently control.

Best, Leonardo

This open letter is my opinion only and not that of my employer or other people I know.

Calculations used

## Mexican immigrants per year
round(-1.7 * 123.166749) * 1000

## [1] -209000

## Percent of Mexican illegal immigrants per year
round(350177 / 679996 * 100)

## [1] 51

## Range of the coyote market: only Mexicans
c(209000, 350000) * 3000 / 1e6

## [1]  627 1050

## Range of the coyote market: everyone
c(209000, 350000) * 3000 / 1e6 * 2

## [1] 1254 2100

## Cost of having 10k more Border Patrol Agents
10000 * 80250 / 1e6

## [1] 802.5

## Cost of having 5k more Border Patrol Agents while increasing their salary by 10%
(5000 * 80250 * 1.1 + 20000 * 80250 * 0.1) / 1e6

## [1] 601.875

## 3000 USD in pesos (rounded to thousands)
round(3 * 21.4230) * 1000

## [1] 64000

## Minimum wage days
round(64000 / 80.04)

## [1] 800

## Minimum wage years -- without a single break!!!
round(800 / 365.25, 1)

## [1] 2.2

I finally wrote this open letter after my poll a few weeks ago:

I'm thinking of writing an open letter to @realDonaldTrump about an alternative solution to the USA-Mexico wall. Would you do it?
— 🇲🇽 Leonardo Collado-Torres (@lcolladotor) January 7, 2017

References

Easily explore a table with shinycsv

Fri, 20 Jan 2017 00:00:00 +0000

Have you ever had to explore a table with data? I believe the answer is yes for most people that work at a computer or even just use it for communicating with their friends and family. Tables of data pop up everywhere, for example in personal finance. Websites like Mint.com allow you to download your transactions in a CSV file called transactions.csv. CSV is one of the many formats for storing tables and most likely when you try to open the transactions.csv file, it will open with Excel. Now, can you make a quick figure of one of your columns in your table?

Some will answer yes, others no. The basic issue is that it’s not super easy to explore your data in Excel or similar programs. Wait, shouldn’t it be easy? 😕

What if you want to subset your data and want to re-make the plot? How about getting some simple statistics like the mean or frequency of some categories for a given variable? 😨 These are some of the immediate tasks that are helpful when exploring data. Visually, making figures with two variables is also very common.

Programmers and experts in Excel, Stata, R among other options can perform these data explorations. It might take them a little bit of time to write the code or remember it or use the user interface menu of their program of choice. But what about everyone else?

At the Lieber Institute for Brain Development where I work, it’s common for us to exchange data in tables, and thus explore data. That’s why we created shinycsv (Collado-Torres, Semick, and Jaffe, 2016). It’s an R package (R Core Team, 2016) that contains a shiny (Chang, Cheng, Allaire, Xie, et al., 2017) application that allows users to interactively explore a table.

Installing R is a pretty high bar, that’s why we are hosting this application at https://jhubiostatistics.shinyapps.io/shinycsv/. Try it out!

shinycsv application

The application includes data about cars to demonstrate what it can do. It’s a small data set that is commonly used for demonstration purposes. Anyhow, in the application you’ll notice a few tabs.

The application shows the raw data in an interactive table that allows you to subset the observations by some criteria, search in the table, and sort in different ways. The raw summary tab shows quick statistical summaries which depend on the variable type (numerical, categorical, etc). If you interacted with the table in raw data then the summaries at raw summary will be based on the subset you selected.

The one variable and two variables tabs are for making figures based on one or two variables at a time. The code in shinycsv tries to guess what’s the best figure for a given type of variable and in case that you are interested in learning R, it also shows the exact code you can use to reproduce the figure in your computer. We added this feature to excite users about learning R. And it’s useful for advanced users too that might want to customize the resulting figures. Hm…, you don’t like the colors we chose for the figure? Well go to plot colors, choose another color, and come back to see your new figure with the color of your choosing. 😄

Hm… but what if you don’t have a CSV file? Well, shinycsv can handle many different tables thanks to rio (Chan, Chan, Leeper, and Becker, 2016). Even Excel sheets! 😉

So, go ahead and test it out! We’ll be glad to hear your feedback at LieberInstitute/shinycsv.

Notes

Note that when I referred to tables earlier, I referred to square tables with different variables (age, height, weight, etc) as columns as observations as rows. That is, Excel files with a single sheet with no comments or figures inside the Excel file.
Are you interested in learning more about R and shiny? Maybe you’ll want to take a look at the showcase mode version of the application.
If you use shinycsv::explore() locally, the file size limit is increased to 500 MB. Although at that point you might want to consider using R or another programming language.
What about casting variables? If you want to have fine control about casting the variables, save your data in a RData file. Sure, this requires an R user.

Reproducibility

## Reproducibility info
library('devtools')
options(width = 120)
session_info()

## Session info -----------------------------------------------------------------------------------------------------------

##  setting  value                                             
##  version  R Under development (unstable) (2016-10-26 r71594)
##  system   x86_64, darwin13.4.0                              
##  ui       X11                                               
##  language (EN)                                              
##  collate  en_US.UTF-8                                       
##  tz       America/New_York                                  
##  date     2017-01-20

## Packages ---------------------------------------------------------------------------------------------------------------

##  package       * version  date       source        
##  bibtex          0.4.0    2014-12-31 CRAN (R 3.4.0)
##  bitops          1.0-6    2013-08-17 CRAN (R 3.4.0)
##  devtools      * 1.12.0   2016-12-05 CRAN (R 3.4.0)
##  digest          0.6.11   2017-01-03 CRAN (R 3.4.0)
##  evaluate        0.10     2016-10-11 CRAN (R 3.4.0)
##  httr            1.2.1    2016-07-03 CRAN (R 3.4.0)
##  knitcitations * 1.0.7    2015-10-28 CRAN (R 3.4.0)
##  knitr         * 1.15.1   2016-11-22 CRAN (R 3.4.0)
##  lubridate       1.6.0    2016-09-13 CRAN (R 3.4.0)
##  magrittr        1.5      2014-11-22 CRAN (R 3.4.0)
##  memoise         1.0.0    2016-01-29 CRAN (R 3.4.0)
##  plyr            1.8.4    2016-06-08 CRAN (R 3.4.0)
##  R6              2.2.0    2016-10-05 CRAN (R 3.4.0)
##  Rcpp            0.12.9   2017-01-14 CRAN (R 3.4.0)
##  RCurl           1.95-4.8 2016-03-01 CRAN (R 3.4.0)
##  RefManageR      0.13.1   2016-11-13 CRAN (R 3.4.0)
##  RJSONIO         1.3-0    2014-07-28 CRAN (R 3.4.0)
##  stringi         1.1.2    2016-10-01 CRAN (R 3.4.0)
##  stringr         1.1.0    2016-08-19 CRAN (R 3.4.0)
##  withr           1.0.2    2016-06-20 CRAN (R 3.4.0)
##  XML             3.98-1.5 2016-11-10 CRAN (R 3.4.0)

References

Citations made with knitcitations (Boettiger, 2015).

[1] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.7. 2015. URL: https://CRAN.R-project.org/package=knitcitations.

[2] C. Chan, G. C. Chan, T. J. Leeper and J. Becker. rio: A Swiss-army knife for data file I/O. R package version 0.4.16. 2016.

[3] W. Chang, J. Cheng, J. Allaire, Y. Xie, et al. shiny: Web Application Framework for R. R package version 1.0.0. 2017. URL: https://CRAN.R-project.org/package=shiny.

[4] L. Collado-Torres, S. Semick and A. E. Jaffe. shinycsv: Explore a table interactively in a shiny application. R package version 0.99.7. 2016. URL: https://github.com/LieberInstitute/shinycsv.

[5] R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. Vienna, Austria, 2016. URL: https://www.R-project.org/.

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

Finding possible class schedules

Tue, 02 Aug 2016 00:00:00 +0000

Over the weekend my brother wanted to figure out his class schedule for the next semester. He is a veterinary medicine and zootechnology student at UNAM. For this upcoming semester there is a set of classes he has to take and each has 8 or so instructor options. The website where he finds the class times lists about 8 pre-constructed class schedules. So he normally finds one he likes quite a bit, and then manually starts checking if he can change X instructor for Y for a given class. He does this based on the referalls and information he has gathered about the instructors, plus he factors in whether it’d be an overall better schedule. For example, he might prefer to have a packed Tuesday if that means that he can leave early on Friday and avoid classes on Saturday.

The problem is that it’s very easy to make a mistake. You (well he) gets all excited thinking that he’s found the perfect schedule. Only to then realize that there is a conflict between two classes. Or that the practical portion of a class is in a location one hour away from the university, meaning that the schedule he has selected won’t work. This process is very frustrating.

I was watching him and I started to think if I could help him with some code. Turns out that it was straightforward to write some code to find which options are valid. Once I wrote a test case, it took us like half an hour to fill out the data. I know that tomorrow is when he and his classmates start registering for classes, so this information might help his classmates.

First, I define some helper functions. These are rather straightforward but I’ll be using them later on. For example, dias() is just there for typing less.

## Helper functions
dias <- function(d, i) {
    paste0(d, i)
}

extract <- function(m, p) {
    m[[p]]
}

extract_names <- function(m, p) {
    names(m)[p]
}

Next comes the input information. I organized it in a set of nested list objects. The schedule is stored as a character vector. For example, Lucia Eliana’s class meets on Wednesdays (__M__iercoles in Spanish) from 9 to 11 am. I only keep the starting hours (9 and 10 am) because otherwise the code won’t detect valid opitons that include another class that starts at 11 am. For classes that are 1 hour away from the university, we included 1 hour before and 1 hour after the class.

## Input class/prof info and schedule
materias <- list(
    repro = list(
        'lucia eliana' = c(dias('m', 9:10), dias('j', 9:10), dias('v', 8:13)),
        'esquivel lacroix' = c(dias('l', 14:15), dias('ma', 12:18), dias('m', 14:15)),
        'ismael porras' = c(dias('l', 9:10), dias('m', 8:14), dias('j', 9:10)),
        'esquivel lacroix 2' = c(dias('l', 14:15), dias('m', 14:15), dias('v', 12:18)),
        'salvador galina' = c(dias('ma', 8:13), dias('j', 9:10), dias('v', 9:10)),
        'alberto balcazar' = c(dias('l', 15:16), dias('j', 12:18), dias('v', 14:15)),
        'ana myriam boeta' = c(dias('l', 8:13), dias('m', 10:11), dias('v', 9:10)),
        'rafael eduardo paz' = c(dias('l', 11:17), dias('j', 14:15), dias('v', 16:17)),
        'juan heberth' = c(dias('ma', 9:10), dias('m', 11:17), dias('v', 11:12)),
        'vicente octavio mejia' = c(dias('ma', 8:17))
    ),
    economia = list(
        'valentin efren espinoza' = c(dias('l', 8:10), dias('ma', 9:11)),
        'maria del pilar velazquez' = c(dias('l', 16:18), dias('m', 16:18)),
        'arturo alonso pesado' = c(dias('l', 11:13), dias('j', 11:13)),
        'laura mendez' = c(dias('ma', 13:15), dias('m', 18:20)),
        'laura mendez 2' = c(dias('j', 11:13), dias('v', 11:13)),
        'manuela garcia' = c(dias('l', 17:19), dias('v', 16:18)),
        'francisco alejandro' = c(dias('ma', 7:9), dias('j', 7:9)),
        'isaac reyes' = c(dias('m', 13:15), dias('v', 13:15)),
        'jose luis tinoco' = c(dias('ma', 12:14), dias('m', 9:11)),
        'isaac reyes 2' = c(dias('l', 14:16), dias('ma', 14:16))
        
    ),
    bacterianas = list(
        'jose luis gutierrez' = dias('s', 8:11),
        'rodrigo mena' = c(dias('ma', 18:19), dias('j', 18:19)),
        'beatriz arellano' = c(dias('l', 7:8), dias('ma', 10:11)),
        'de la pena, ramirez ortega' = c(dias('j', 18:19), dias('v', 18:19)),
        'ramirez ortega' = c(dias('m', 7:8), dias('j', 7:8)),
        'rodrigo mena 2' = c(dias('ma', 16:17), dias('m', 16:17)),
        'de la pena' = dias('s', 8:11),
        'efren diaz aparicio' = dias('s', 8:11),
        'lucia del carmen favila' = dias('s', 8:11)
    ),
    parasitarias = list(
        'cintli martinez' = c(dias('j', 16:17), dias('v', 18:20)),
        'osvaldo froylan' = c(dias('ma', 18:19), dias('j', 18:20)),
        'maria quintero, agustin perez' = c(dias('ma', 13:14), dias('m', 7:9)),
        'maria quintero' = c(dias('m', 16:18), dias('j', 16:17)),
        'evangelina romero' = c(dias('ma', 7:8), dias('v', 7:9)),
        'guadarrama 01' = c(dias('m', 7:8), dias('j', 11:13)),
        'guadarrama 03' = c(dias('ma', 13:15), dias('v', 7:8)),
        'guadarrama 04' = c(dias('l', 16:17), dias('ma', 18:20)),
        'guadarrama 05' = c(dias('l', 7:9), dias('j', 7:8))
    ),
    diagnosticas = list(
        '1701' = c(dias('l', 11:13), dias('m', 11:16)),
        '1702' = c(dias('j', 13:15), dias('v', 13:18)),
        '1703' = c(dias('ma', 7:9), dias('v', 8:13)),
        '1704' = c(dias('l', 18:20), dias('j', 13:18)),
        '1705' = c(dias('l', 11:13), dias('m', 7:11)),
        '1706' = c(dias('ma', 15:17), dias('m', 15:19)),
        '1707' = c(dias('ma', 10:12), dias('j', 10:15)),
        '1708' = c(dias('l', 18:20), dias('ma', 15:19)),
        '1709' = c(dias('l', 11:13), dias('j', 8:13)),
        '1711' = c(dias('j', 13:15), dias('v', 10:13))
    )
)

Now that the input information is complete, I use expand.grid() to find out all the different possible options.

## Get all the options
options <- expand.grid(lapply(materias, function(x) { seq_len(length(x))}))
dim(options)

## [1] 81000     5

There’s 81,00 of them including the classes that meet on Saturday. You can see why it’s a frustrating process to find which combination of classes work when doing this manually.

Next, I explore all these options to find those that are valid, meaning that none of the classes overlap. I do this by finding which options have no duplicated hours from the character vectors defined earlier. Nothing fancy.

valid <- apply(options, 1, function(input) {
    info <- mapply(extract, materias, input)
    !any(duplicated(unlist(info)))
})

Now that I have the valid options, I can find the names of the instructors for them. There’s 2,847 valid schedules in the end, out of the 81,000. That’s 3.5 percent!

valid_prof <- apply(options[valid, ], 1, function(input) {
    mapply(extract_names, materias, input)
})
ncol(valid_prof)

## [1] 2847

You can search the interactive version here to select only the options with a given instructor. For example, in my brother’s case there are 30 valid options once he decided to prioritize two instructors as shown in the non-interactive table below.

## Ideally, this code would create an interactive table, but it doesn't work for some reason:
#library('DT')
#datatable(t(valid_prof), options = list(pagingType='full_numbers', pageLength=10), rownames = FALSE)
valid_prof <- t(valid_prof)
rownames(valid_prof) <- seq_len(nrow(valid_prof))
top_options <- valid_prof[valid_prof[, 1] == 'lucia eliana' & valid_prof[, 2] %in% c('isaac reyes', 'isaac reyes 2'), ]
kable(top_options, format = 'markdown', row.names = TRUE)

	repro	economia	bacterianas	parasitarias	diagnosticas
5	lucia eliana	isaac reyes 2	jose luis gutierrez	cintli martinez	1701
11	lucia eliana	isaac reyes 2	rodrigo mena	cintli martinez	1701
14	lucia eliana	isaac reyes 2	beatriz arellano	cintli martinez	1701
19	lucia eliana	isaac reyes 2	ramirez ortega	cintli martinez	1701
25	lucia eliana	isaac reyes 2	de la pena	cintli martinez	1701
31	lucia eliana	isaac reyes 2	efren diaz aparicio	cintli martinez	1701
37	lucia eliana	isaac reyes 2	lucia del carmen favila	cintli martinez	1701
46	lucia eliana	isaac reyes 2	jose luis gutierrez	osvaldo froylan	1701
50	lucia eliana	isaac reyes 2	beatriz arellano	osvaldo froylan	1701
58	lucia eliana	isaac reyes 2	ramirez ortega	osvaldo froylan	1701
67	lucia eliana	isaac reyes 2	de la pena	osvaldo froylan	1701
76	lucia eliana	isaac reyes 2	efren diaz aparicio	osvaldo froylan	1701
85	lucia eliana	isaac reyes 2	lucia del carmen favila	osvaldo froylan	1701
123	lucia eliana	isaac reyes 2	jose luis gutierrez	guadarrama 01	1701
130	lucia eliana	isaac reyes 2	rodrigo mena	guadarrama 01	1701
134	lucia eliana	isaac reyes 2	beatriz arellano	guadarrama 01	1701
137	lucia eliana	isaac reyes 2	de la pena, ramirez ortega	guadarrama 01	1701
144	lucia eliana	isaac reyes 2	de la pena	guadarrama 01	1701
151	lucia eliana	isaac reyes 2	efren diaz aparicio	guadarrama 01	1701
158	lucia eliana	isaac reyes 2	lucia del carmen favila	guadarrama 01	1701
209	lucia eliana	isaac reyes 2	jose luis gutierrez	guadarrama 05	1701
217	lucia eliana	isaac reyes 2	rodrigo mena	guadarrama 05	1701
222	lucia eliana	isaac reyes 2	de la pena, ramirez ortega	guadarrama 05	1701
232	lucia eliana	isaac reyes 2	de la pena	guadarrama 05	1701
242	lucia eliana	isaac reyes 2	efren diaz aparicio	guadarrama 05	1701
252	lucia eliana	isaac reyes 2	lucia del carmen favila	guadarrama 05	1701
872	lucia eliana	isaac reyes 2	jose luis gutierrez	guadarrama 05	1704
885	lucia eliana	isaac reyes 2	de la pena	guadarrama 05	1704
894	lucia eliana	isaac reyes 2	efren diaz aparicio	guadarrama 05	1704
903	lucia eliana	isaac reyes 2	lucia del carmen favila	guadarrama 05	1704

Reproducibility

## Reproducibility info
library('devtools')
session_info()

## Session info --------------------------------------------------------------

##  setting  value                       
##  version  R version 3.3.0 (2016-05-03)
##  system   x86_64, mingw32             
##  ui       RStudio (0.99.902)          
##  language (EN)                        
##  collate  English_United States.1252  
##  tz       America/Mexico_City         
##  date     2016-08-02

## Packages ------------------------------------------------------------------

##  package   * version date       source        
##  devtools  * 1.12.0  2016-06-24 CRAN (R 3.3.1)
##  digest      0.6.9   2016-01-08 CRAN (R 3.3.0)
##  evaluate    0.9     2016-04-29 CRAN (R 3.3.0)
##  formatR     1.4     2016-05-09 CRAN (R 3.3.0)
##  highr       0.6     2016-05-09 CRAN (R 3.3.0)
##  knitr     * 1.13    2016-05-09 CRAN (R 3.3.0)
##  magrittr    1.5     2014-11-22 CRAN (R 3.3.0)
##  memoise     1.0.0   2016-01-29 CRAN (R 3.3.0)
##  rsconnect   0.4.3   2016-05-02 CRAN (R 3.3.0)
##  stringi     1.1.1   2016-05-27 CRAN (R 3.3.0)
##  stringr     1.0.0   2015-04-30 CRAN (R 3.3.0)
##  withr       1.0.2   2016-06-20 CRAN (R 3.3.1)

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

Federico Sánchez Rodríguez 1950-2016

Mon, 04 Apr 2016 00:00:00 +0000

Today the UNAM community at large mourns the passing of Federico Sánchez Rodríguez. He got his bachelor’s degree from the School of Chemistry - UNAM, masters and PhD degrees from Biomedicas - UNAM, postdoc from UCSF, was a member of CIFN-UNAM now called CCG-UNAM (it’s his affiliation in this 1983 paper), and worked most of his career at IBT-UNAM.

I’m sure that he made many friends, trained many students at all levels, and had a highly productive academic career as evidenced on his homepage where he lists many papers, patents, etc. A PubMed author search includes his papers but be careful to not confuse him with other authors that shared his name.

I’ll remember Federico fondly for the time we shared at LCG-UNAM. He was a great teacher and motivated me to ask as many questions as I could think of. My background in biology was weaker than my classmates, and I loved to ask questions of the sort: if X biolgical process is possible, could Y happen in the cell? He let my imagination run wild. Federico was very supportive of an elective class a few of us organized in our 4th year. You could always tell that he fed off the energy of enthusiastic students. He was always there if you needed some advice. At the end of my time at LCG-UNAM, he enjoyed how I argued against other members of the LCG academic committee. He always supported me in my non-traditional choices. Federico knew many things about plants, but also about the best food, the best drinks, and was always eager to share his knowledge with you.

In one of my last interactions with him at FedeFest in August 2014, I asked him a question in the style that I used to during class. I asked him how would he evaluate results from single-cell sequencing and verify that the results were indeed biogical and not technical. A month later, I sent him this paper.

Fede, I will always remember you fondly.

Best, Leonardo

LCG-UNAM 2005-2009

LCG-UNAM 11 years ago

Thu, 10 Mar 2016 00:00:00 +0000

This has been a busy week. I just flew in last night from #ENAR2016 and I’m getting ready for a postdoc interview tomorrow, which means that I’m also flying today. On the flight back from #ENAR2016 I started reading a book my friend John Muschelli bought for me. It’s called The Happiness Advantage by Shawn Anchor and so far I’m loving it. There are several things that it talks about that I’ve done in the past, but maybe not in the past year. Anyhow, I’ll probably talk about it in a separate blog post.

But coincidentally, a friend of mine from undergrad just shared with me the following YouTube video. Watching it made me very happy! We definitely had a blast during undergrad. The person that talks about Boston is Frederick Bieber (he’s kinda hard to google due to his last name) who at the time worked with 5 of my classmates.

My undergrad class from LCG-UNAM (link is in English) was the 3rd one (yup, still guinea pigs) and the only one to start off with 40 students. Some left to study other degrees, most of us continue to work in academia getting our master’s, PhDs, postdocs done; others have chosen different career paths, and sadly one of us already passed away. But I bet we can all say that 2005-2009 were good and happy years.

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

Are you doing parallel computations in R? Then use BiocParallel

Mon, 07 Mar 2016 00:00:00 +0000

It’s the morning of the first day of oral conferences at #ENAR2016. I feel like I have a spidey sense since I woke up 3 min after an email from Jeff Leek; just a funny coincidence. Anyhow, I promised Valerie Obenchain at #Bioc2014 that I would write a post about one of my favorite Bioconductor packages: BiocParallel (Morgan, Obenchain, Lang, and Thompson, 2016). By now it’s on the top 5% of downloaded Bioconductor packages, so many people know about it or are unaware that their favorite package uses it behind the scenes.

While I haven’t blogged about BiocParallel yet, I did give a presentation about it at our computing club back in April 2nd, 2015. See it here (source). I’m going to follow its structure in this post.

Parallel computing

Before even thinking about using BiocParallel you have to decide whether parallel computing is the thing you need.

While I’m not talking about cloud computing, I still find this picture funny.

There’s different types of parallel computing, but what I’m referring to here is called embarrassingly parallel where you have a task to do for a set of inputs, you split your inputs into subsets and perform the task on these subsets. Performing this task for one input a a time is called serial programming and it’s what we do in most cases when using functions like lapply() or for loops.

plot(y = 10 / (1:10), 1:10, xlab = 'Number of cores', ylab = 'Time',
    main = 'Ideal scenario', type = 'o', col = 'blue',
    cex = 2, cex.axis = 2, cex.lab = 1.5, cex.main = 2, pch = 16)

You might be running a simulation for a different set of parameters (a parameter grid) and running each simulation could take some time. Parallel computing can help you speed up this problem. In the ideal scenario, the higher number of computing cores (units that evaluate subsets of your inputs) the less time you need to run your full analysis.

plot(y = 10 / (1:10), 1:10, xlab = 'Number of cores', ylab = 'Time',
    main = 'Reality', type = 'o', col = 'blue',
    cex = 2, cex.axis = 2, cex.lab = 1.5, cex.main = 2, pch = 16)
lines(y = 10 / (1:10) * c(1, 1.05^(2:10) ), 1:10, col = 'red',
    type = 'o', cex = 2)

However, in reality parallel computing is not cost-free. It involves some communication costs, like sending the data to the cores, aggregating the results in a way that you can then easily use, among other things. So, it’ll be a bit slower than the ideal scenario but you can potentially still greatly reduce the overall time.

Having said all of the above, lets say that you now want to do some parallel computing in R. Where do you start? A pretty good place to start is the CRAN Task View: High-Performance and Parallel Computing with R. There you’ll find a lot of information about different packages that enable you to do parallel computing with R.

But you’ll soon be lost in a sea of new terms.

Why use BiocParallel?

It’s simple to use.
You can try different parallel backends without changing your code.
You can use it to submit cluster jobs.
You’ll have access to great support from the Bioconductor developer team.

Those are the big reasons of why I use BiocParallel. But let me go through them a bit more slowly.

Birthday example

I’m going to use as an example the birthday problem where you want to find out empirically the probability that two people share the same birthday in a room.

birthday <- function(n) {
    m <- 10000
    x <- numeric(m)
    for(i in seq_len(m)) {
        b <- sample(seq_len(365), n, replace = TRUE)
        x[i] <- ifelse(length(unique(b)) == n, 0, 1)
    }
    mean(x)
}

Naive birthday code

Once you have written the code for it, you can then use lapply() or a for loop to calculate the results.

system.time( lapply(seq_len(100), birthday) )

##    user  system elapsed 
##  25.610   0.442  27.430

Takes around 25 seconds.

Via doMC

If you looked at CRAN Task View: High-Performance and Parallel Computing with R you might have found the doMC (Analytics and Weston, 2015).

It allows you to run computations in parallel as shown below.

library('doMC')

## Loading required package: foreach

## Loading required package: iterators

## Loading required package: parallel

registerDoMC(2)
system.time( x <- foreach(j = seq_len(100)) %dopar% birthday(j) )

##    user  system elapsed 
##  12.819   0.246  13.309

While it’s a bit faster, the main problem is that you had to change your code in order to be able to use it.

With BiocParallel

This is how you would run things with BiocParallel.

library('BiocParallel')
system.time( y <- bplapply(seq_len(100), birthday) )

##    user  system elapsed 
##   0.021   0.011  16.095

The only change here is using bplapply() instead of lapply(), so just 2 characters. Well, that and loading the BiocParallel package.

BiocParallel’s advantages

There are many computation backends and one of the strongest features of BiocParallel is that it’s easy to switch between them. For example, my computer can run the following options:

registered()

## $MulticoreParam
## class: MulticoreParam 
##   bpjobname:BPJOB; bpworkers:2; bptasks:0; bptimeout:Inf; bpRNGseed:; bpisup:FALSE
##   bplog:FALSE; bpthreshold:INFO; bplogdir:NA
##   bpstopOnError:FALSE; bpprogressbar:FALSE
##   bpresultdir:NA
## cluster type: FORK 
## 
## $SnowParam
## class: SnowParam 
##   bpjobname:BPJOB; bpworkers:2; bptasks:0; bptimeout:Inf; bpRNGseed:; bpisup:FALSE
##   bplog:FALSE; bpthreshold:INFO; bplogdir:NA
##   bpstopOnError:FALSE; bpprogressbar:FALSE
##   bpresultdir:NA
## cluster type: SOCK 
## 
## $SerialParam
## class: SerialParam 
##   bplog:FALSE; bpthreshold:INFO
##   bpcatchErrors:FALSE

If I was doing this in our computing cluster, I would see even more options.

Now lets say that I want to test different computation backends, or even run things in serial mode so I can trace a bug down more easily. Well, all I have to do is change the BPPARAM argument as shown below.

## Test in serial mode
system.time( y.serial <- bplapply(1:10, birthday,
    BPPARAM = SerialParam()) )

##    user  system elapsed 
##   2.577   0.033   2.733

## Try Snow
system.time( y.snow <- bplapply(1:10, birthday, 
    BPPARAM = SnowParam(workers = 2)) )

##    user  system elapsed 
##   0.027   0.006   2.436

Talking about computing clusters, you might be interested in using BatchJobs (Bischl, Lang, Mersmann, Rahnenführer, et al., 2015) just like Prasad Patil did for his PhD work. Well, with BiocParallel you can also chose to use the BatchJobs backend. I have code showing this at the presentation I referenced earlier.

Where do I start?

If you are convinced about using BiocParallel, which I hope you are by now, check out the Introduction to BiocParallel vignette available at BiocParallel’s landing page. It explains in more detail how to use it and it’s rich set of features. But if you just want to jump right in and start playing around with it, install it by running the following code:

install.packages("BiocManager")
BiocManager::install("BiocParallel")

Conclusions

Like I said earlier, BiocParallel is simple to use and has definite advantages over other solutions.

You can try different parallel backends without changing your code.
You can use it to submit cluster jobs.
You’ll have access to great support from the Bioconductor developer team. See the biocparallel tag at the support website.

Have fun using it!

Reproducibility

## Reproducibility info
library('devtools')
session_info()

## Session info --------------------------------------------------------------

##  setting  value                       
##  version  R version 3.2.2 (2015-08-14)
##  system   x86_64, darwin13.4.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  tz       America/Chicago             
##  date     2016-03-07

## Packages ------------------------------------------------------------------

##  package        * version  date       source        
##  bibtex           0.4.0    2014-12-31 CRAN (R 3.2.0)
##  BiocParallel   * 1.4.3    2015-12-16 Bioconductor  
##  bitops           1.0-6    2013-08-17 CRAN (R 3.2.0)
##  codetools        0.2-14   2015-07-15 CRAN (R 3.2.2)
##  devtools       * 1.10.0   2016-01-23 CRAN (R 3.2.3)
##  digest           0.6.9    2016-01-08 CRAN (R 3.2.3)
##  doMC           * 1.3.4    2015-10-13 CRAN (R 3.2.0)
##  evaluate         0.8      2015-09-18 CRAN (R 3.2.0)
##  foreach        * 1.4.3    2015-10-13 CRAN (R 3.2.0)
##  formatR          1.2.1    2015-09-18 CRAN (R 3.2.0)
##  futile.logger    1.4.1    2015-04-20 CRAN (R 3.2.0)
##  futile.options   1.0.0    2010-04-06 CRAN (R 3.2.0)
##  httr             1.1.0    2016-01-28 CRAN (R 3.2.3)
##  iterators      * 1.0.8    2015-10-13 CRAN (R 3.2.0)
##  knitcitations  * 1.0.7    2015-10-28 CRAN (R 3.2.0)
##  knitr          * 1.12.3   2016-01-22 CRAN (R 3.2.3)
##  lambda.r         1.1.7    2015-03-20 CRAN (R 3.2.0)
##  lubridate        1.5.0    2015-12-03 CRAN (R 3.2.3)
##  magrittr         1.5      2014-11-22 CRAN (R 3.2.0)
##  memoise          1.0.0    2016-01-29 CRAN (R 3.2.3)
##  plyr             1.8.3    2015-06-12 CRAN (R 3.2.1)
##  R6               2.1.2    2016-01-26 CRAN (R 3.2.3)
##  Rcpp             0.12.3   2016-01-10 CRAN (R 3.2.3)
##  RCurl            1.95-4.7 2015-06-30 CRAN (R 3.2.1)
##  RefManageR       0.10.6   2016-02-15 CRAN (R 3.2.3)
##  RJSONIO          1.3-0    2014-07-28 CRAN (R 3.2.0)
##  snow             0.4-1    2015-10-31 CRAN (R 3.2.0)
##  stringi          1.0-1    2015-10-22 CRAN (R 3.2.0)
##  stringr          1.0.0    2015-04-30 CRAN (R 3.2.0)
##  XML              3.98-1.3 2015-06-30 CRAN (R 3.2.0)

References

Citations made with knitcitations (Boettiger, 2015).

[1] R. Analytics and S. Weston. doMC: Foreach Parallel Adaptor for 'parallel'. R package version 1.3.4. 2015. URL: http://CRAN.R-project.org/package=doMC.

[2] B. Bischl, M. Lang, O. Mersmann, J. Rahnenführer, et al. “BatchJobs and BatchExperiments: Abstraction Mechanisms for Using R in Batch Environments”. In: Journal of Statistical Software 64.11 (2015), pp. 1–25. URL: http://www.jstatsoft.org/v64/i11/.

[3] C. Boettiger. knitcitations: Citations for 'Knitr' Markdown Files. R package version 1.0.7. 2015. URL: http://CRAN.R-project.org/package=knitcitations.

[4] M. Morgan, V. Obenchain, M. Lang and R. Thompson. BiocParallel: Bioconductor facilities for parallel evaluation. R package version 1.4.3. 2016.

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

Teaching a short topic to beginner R users

Tue, 07 Jul 2015 00:00:00 +0000

A couple weeks ago I was given the opportunity to teach a 1 hr 30 min slot of an introduction to R course. In the past, I’ve taught lectures for similar courses, and I ended up asking myself what would be the best short topic to teach and how to teach it.

Best short topic

There are two ways to answer the first question, one boring and one more interesting. The boring answer is that the course instructor selected the topic. The interesting one goes like this. I have taken short R courses before and taught others, and it’s always overwhelming for the students. You get to cover many concepts, get familiarized with R’s syntax, and in the end without lots of practice it’s very challenging to retain much of the information. I think that students love it when they learn how to do something simple that could be the first building block for many of their projects. In parallel, I think that one of the coolest R topics you can learn in an hour is how to create reproducible documents with rmarkdown (Allaire, Cheng, Xie, McPherson, et al., 2015).

Learning how to use a single function, render() in this case, is as simple as it gets. And using the RStudio Desktop is even simpler. Of course, it can easily get complicated. For example, on a new computer you need to install all the LaTeX dependencies if you want to create PDF files. That task can take some time and maybe scare away some new users. But PDF files are really a plus in this case since you can start creating HTML and Word documents. Other complications arise when a user is interested in more control over formatting the file, but like I said earlier, all you need is a simple building block and rmarkdown is clearly one of them.

This is why the final answer to the first question was teaching how to use rmarkdown to create reproducible reports (HTML, Word files) using R.

How to teach it

Teaching a short topic to a beginner’s audience is no easy feat. In the past I’ve made lectures that have the code for every single step and many links to resources where students can learn some details. That is, I’ve created the lectures in such a way that a student can later use them as reference and follow them without an instructor explaining them.

That’s a strategy that I think works on the long run. However, it makes the actual lecture boring and very limited in interactivity. At the JHSPH biostat computing club, other students have chosen to use a lot of images, funny to witty quotes, and asked listeners to voice their opinions. I’ve come to enjoy those presentations and I decided to create my lecture following that trend.

I started off with a series of questions about reproducible research and asked students to voice their opinions and to define a few key concepts. A couple were aware of the difference between reproducibility and replicability, but most were not. I also questioned them and presented them verbally with some famous cases, so they could realize that it’s a fairly complicated matter. Next I presented some answers and definitions from the Implementing Reproducible Research book.

Specifically talking about R, I showed the students several documents I’ve created in the past and asked whether they thought that they could reproduce the results or not. Basically, I wanted to highlight that when using R, you really need the session information if you want to reproduce something. Specially if the analysis involves packages under heavy development.

After having motivating the need for reproducible documents, I briefly showed what rmarkdown is with some images from RStudio shown below.

That gave the students a general idea of how these documents look when you are writing them. But the most important part was showing them examples of how the resulting documents look like. That is, I showed them some complicated projects so they could imagine doing one themselves. The examples included some books, but given the audience I think that the one that motivated them most was Alyssa Frazee’s polyester reproducible paper (check the source here). I also showed them some of the cool stuff you can create with HTML documents: basically adding interactive elements.

From there, we left the presentation and I demo’ed how to use RStudio to write rmarkdown documents, the Markdown syntax, where to find help, etc.

By this point, I think the lecture was quite complete and the students were motivated. However, from my past experience, I’ve come to realize that students will easily forget a topic if they don’t practice doing it. That is why even before making the lecture I spent quite a bit of time designing two practice labs. Both labs involved creating a rmarkdown document.

The first lab included some cool illusion plots which involved a lot of R code. The code wasn’t the point, but simply learning some of the basics such as what is a code chunk, some of Markdown’s syntax, specifying some code chunk options, adding the session information, and using inline R code to show the date when the document was made. Ahh, and of course, uploading your HTML document to RPubs (see mine). I know that not everyone is a fan of RPubs, but I imagined that students would get super excited that they made something that they could then show their colleagues and friends. And some did!

Sadly, we didn’t have enough time for the second lab. I did explain to the students what it was about, but they didn’t have time to do it themselves. For this second document, I wanted the students to learn how to create a document reporting some results where all the numbers in the text are written by R instead of copy-pasting them.

Conclusions

As you can see, I enjoyed thinking what to teach and specially how to teach a short topic to beginner R students. Thanks to having one of the later sessions, I could teach them how to use rmarkdown in a way that hopefully left them highly motivated to do it themselves. I hope that most of them will take that they learned in that module and others and apply them in their day to day work.

References

You can find the lecture itself here but like I said earlier, it was designed for class and not for being used as a reference. However, the lab and it’s key might be more useful.

Citations made with knitcitations (Boettiger, 2015).

[1] J. Allaire, J. Cheng, Y. Xie, J. McPherson, et al. rmarkdown: Dynamic Documents for R. R package version 0.7. 2015. URL: http://CRAN.R-project.org/package=rmarkdown.

[2] C. Boettiger. knitcitations: Citations for Knitr Markdown Files. R package version 1.0.6. 2015. URL: http://CRAN.R-project.org/package=knitcitations.

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

What are reasonable work hours and salary expectations for biostats/genomics careers?

Thu, 12 Feb 2015 00:00:00 +0000

My advisor recently asked me to fill a career planning document (he’ll blog about it at some point) that has lots of questions and is being very useful. Trying to fill answer all these questions has gotten me thinking about other important things for the future. Two of them being reasonable expectations for work hours and salary in a biostats/genomics academic career.

I find these questions hard to answer and even hard to talk about, and if you know me, I’m a person that asks lots of questions. I have also been advised against talking about these subjects during a first interview because it’s important to focus on the research environment. I agree that you should find a place to work that you like with problems that will keep you motivated, and with colleagues that make your daily life enjoyable (Hall, 2015).

However, I still want to have a reasonable idea about what to expect. Given that it’s hard to ask these questions, it’s also hard to blog about it. These questions are not really what others encourage you to talk about (Gould, 2015; Gould, 2014) but I think that I’m not the only one with these questions in mind and that others might benefit from this post. Well, I hope they will.

I didn’t ask myself these questions before doing a PhD. It felt natural that doing a PhD would open the door for more opportunities in academia than going to work after undergrad or a masters. Even for a career outside science which a bit more than half (53%) of UK science PhD students choose to follow (Gould, 2014), getting a PhD likely gives a strong boost to your career options. But when you start considering a postdoc, there is an opportunity cost (, 2015) associated to it because you could be earning more at a non-academic job. But how much are we talking about?

Picture from “The postdoc decision” (Gould, 2014).

Regardless of the opportunity cost of doing a postdoc, for a career in academia you will likely want to do a postdoc to get a chance at a tenure-track job, specially in fields like biology. For a career outside academia but still research related, I think that doing a postdoc could increase your chances at getting a better job. In any case, you have to consider all career options when doing a postdoc (, 2014).

Salary: what to expect

For a postdoc, the NIH sets training stipends which sets a baseline (, 2015). For example, 42k per year in 2014.

For later, ASA: Career Resources for Statisticians has information for biostatistics departments. For example, the median year 0 assistant professor in a biostatistics department made 102k (megan, 2014). From this survey data you can get an idea of the salary progression through your academic career.

Similarly, I find the Life Sciences Salary Survey (2013, 2014) informative. In the 2014 detailed pdf you can see that a male postdoc average pay was $46,372, $79,616 for an assistant professor, $107,868 for associate professors, and $158,059 for professors. That last figure is smaller than $169,900 which is the median salary for year 0 professors in biostatistics (megan, 2014). The 2014 Life Sciences survey also has average salaries by age. So you can make three different progressions of your career salary.

Why should you care?

There are probably many reasons why you should care about your salary given two or more equally good work opportunities. In my case, I wanted to get a rough idea if an academic career would be able to pay for my desired lifestyle and life goals. Let me explain this in more detail.

From looking at the retirement for doctoral scientists report (Foundation, 2015) retiring between 66 and 70 years old seems like a common scenario. Then from the life expectancy calculator (OACT, 2015), it seems that for someone like me the average life expectancy is 87 years and retiring from work at 67 years. That is, about 37 years of work and 20 years of retirement. Then, the average cost of a house is about 320k (McBride, 2014). If you get a house loan for 270k at 4% interest rate for 30 years and 1.25% property tax you would pay 190k in interest and 120k in tax (Unknown, 2015). Federal income tax is about 28% (, 2015) (I know it’s more complicated) with about 5% for state tax (, 2015). So about 33% of your income goes to paying taxes. The cost for having a car is about 10k a year (Larry Copeland, 2013) although you could probably save some money by getting a smaller car.

I think that I want to have children in the future, and if I do, I would like to afford their education. It seems that the cost of raising a child to age 18 is about 575k if you spend about 120k in private college (, 2015). Those numbers include the cost of adapting your house (but not interests in mortgage).

I also want to be able to travel quite a bit, and given that these are all US numbers, I would also like to go to Mexico frequently. Anyhow, setting about 5k a year for traveling is an ambitious goal. Alternatively, the 5k could help mitigate any unforeseen emergency, like a surgery.

I can use the numbers from all three academic career salary progressions to get an idea of how my life goals fit in and the impact they would have on my lifestyle. For the average salary for life sciences by age I’m using the numbers from 30 to 67 years old. For average salary for life sciences by career stage I’ll use 2 years for postdoc, 7 years for assistant professor, 8 for associate professor and the remaining 20 as a professor. For the progression for biostatistics departments in research universities I simply adapted the above numbers to the greater data resolution available.

Looking at my mint.com budget I see that I spend about 1370 a month on everything except rent. I like my current lifestyle and would like to keep it throughout my life, obviously adapting it to my needs.

Monthly difference between costs for idealized life and current life style (1370 a month) using different salary progression data. That is, how much you can save per month for other misc expenses.
	NoChild	OneChild	TwoChild
average	316	-525	-1366
tenure	1037	197	-644
biostats	2263	1422	581

Now, in the previous table only the life sciences tenure and biostats progressions can handle my life goals without sacrificing my life style if I had one child. Two children only seems feasible in the biostats salary progression. These numbers give a general idea, but it’s way more information than I had before.

Note that I’m assuming that your savings can keep up with inflation. Plus, I don’t have student loans thanks to receiving my education at UNAM-Mexico, but I know that this isn’t true for many US citizens. You could probably do a bit better than I projected if you invested and got a higher return than inflation, got promoted faster, moved to a country with a lower cost of living after saving some money, or had to borrow less money for buying the house. This last point could certainly be feasible if you managed to earn more money early in your career.

Work hours: what to expect

Answering this question has been much harder than the salary question. First, lets look outside of science. A recent news article (, 2015) says: “About 22 percent of Japanese work more than 49 hours a week, compared with 16 percent of Americans, 11 percent of the French and Germans, according to data compiled by the Japanese government.”

A different survey shows that 39 percent of full time employed US workers work beyond 49 hours per week with the average being 47 hours (@jenamcgregor, 2014). Broadly summarized government data shows lower averages (, 2015). Anyhow, I think that it’s likely that work hours for researchers are in the upper half. I’m also under the impression that most scientists work beyond the official working time (check wiki).

I also agree with the idea that work hours fluctuate, for example with weeks when grants are due reaching high peaks. Also, probably a lot of people haven’t tracked their hours (@DynamicEcology, 2014). In my case, using RescueTime I know that I have spent about 45-50 hours per week on my work computer recently (I only have the free version, so I can’t go back too far in time). Though I do get distracted at times and my “productivity pulse” is around 60 to 70.

I certainly don’t think that scientists should be expected to routinely work 60+ hours a week. There’s even a 1943 (wartime) paper arguing that “For men a working week in excess of 60 hours usually leads to no increase in production, and for women the hours should never exceed 54” (see the pdf). But teaching loads can certainly push to higher work loads as this particular story shows (, 2013).

At this point I’m thinking that 45-50 hours per week should be reasonable with a good work-life balance.

But then, “postdocs should expect to work longer-than-average hours” and “We put in the hours we do because it is highly competitive out there” (Gould, 2011).

On the positive side, it seems that research could be a 9-5 job and achieving it “requires good planning, effective teamwork and a supportive environment” (Gould, 2014). This ties in with some of my thoughts: I feel like it’s all about being efficient as you can be and indeed having a great work team. So yes, when looking for a job it is very important to find a job with the right research and support environment.

Why should you care?

I think that it’s important to know this information so you can have realistic expectations of your stress level and work-life balance. If you underestimate the work load, you’ll probably be uncomfortable to say the least.

I also think that it’s dangerous to ignore this question even when you are highly motivated by your work and easily lose track of time spent working. That is, I assume that many scientists are motivated by their work but I argue that they should still think about their life outside work.

Alternative way to count

Picture from the Sleep Report (, 2019 update).

Another way to take a look at this question is to see how would you like to spend your time outside work. Seems like a good recommendation is to sleep 8 hours per day. Spending about 4 hours per day with your family seems like a good goal (, 2015) although many people can’t (, 2015) despite “spending quality family time together is very important” (, 2015). If you exercise about 1 hour a day, take about 1 hour to get ready in the morning, spend 1 hour commuting, and 1 hour cooking/having dinner that leaves you with 8 hours per day for work. With school taking about 6 hours, this seems doable if your work allows you to work from home for 2 hours or so. Such a schedule would look something like this:

Wake up at 6:30, exercise 1 hour
Get ready: 1 hour
Commute: 30 min. You get to work at 9 am.
Work 6 hours until 3pm.
Commute: 30 min
Family time 2 hrs
Dinner 1 hour
Family time 2 hrs
Work 2 more hours: 8:30 to 10:30pm
Sleep 8 hours

Now that I think about it, it seems like my advisor is doing something along these lines. Well, he is likely sleeping much less.

Following this type of workday, you would have to work 5-10 hours on the weekend to get to 45-50 work hours per week. Seems doable and reasonable to me. Working beyond 50 hours a week would most likely mean that you’d have to sacrifice family and exercise time. That doesn’t seem healthy but I understand that it might be needed at times.

This is when ideas that at first might seem unreasonable start to make some sense. For example, James W. Vaupel suggested working 25 hours a week and retiring later in life so young people get more time with their kids (, 2015).

Conclusions

Learning what to expect in terms of salary and work hours for a biostats/genomics career in academia is challenging and time consuming. At best you depend on surveys which may be biased, information from commercial sites and blog posts. However, with salary progression tables you can get a broad idea of how realistic your life goals and lifestyle expectations are. Although it would be great to have more data and data from other countries. For example, I don’t have and haven’t looked for the salary progression data in Mexico: maybe salaries are lower but due to lower costs of living you might be better off there than in the US assuming that the research environment is as good or better. That is a question for another time.

I have found some answers which I hope will be useful later on. From the salary perspective, I find it motivating that this career could fund my goals and lifestyle, specially in a Biostatistics department from a research university. From the work hours perspective, at least I have some information to support my previously vague idea that 45-50 hours a week is reasonable. Whether that’s enough is yet to be seen. Specially with how competitive it can be to get an academic position and complete the progression to becoming a professor.

But even if tools like the PI predictor (, 2015) say that I have a 10% chance of becoming a PI (data up to nov 2014), I have encountered that all my life: 10% of LCG applicants who passed UNAM’s exam got into my undergrad, 10% of applicants got into my PhD program. So it’s old news and I just have to keep working to beat the odds.

Reproducibility

Code for making the table:

life_style <- function(progression = 'average', child = 1, tax = .33, retiredyrs = 20, childcost = 575e3, house = 32e4, interest = 19e4, proptax = 12e4, car = 1e4, travel = 5e3) {
    if(progression == 'average') {
        ## Using http://www.the-scientist.com/Nov2014/11_14_SalarySurveyforWeb.pdf
        ## 2014 average compensation in the US for life scientists by age
        sal <- c(58688, 73577, 93946, 108241, 122558, 130551, 139525, 150995)
        yrs <- rep(c(5, 2), c(7, 1))
    } else if (progression == 'tenure') {
        ## Using http://www.the-scientist.com/Nov2014/11_14_SalarySurveyforWeb.pdf
        ## 2014 average male compensation by career stage in academia
        sal <- c(46372, 79616, 107868, 158059)
        yrs <- c(2, 7, 8, 20)
    } else if (progression == 'biostats') {
        ## Using http://magazine.amstat.org/blog/2014/01/01/academic-salary-survey-2/
        ## Work in biostatistics deparments at a research institution
        ## Using life sciences male postdoc average salary as this data is 
        ## missing in this survey.
        sal <- c(46372, 102200, 104000, 105100, 108300, 124700, 129400, 131000, 129700, 169900, 175300, 193400, 235100, 207700, 207900)
        yrs <- c(2, 1, 2, 2, 2, 1, 2, 3, 2, 1, 3, 3, 3, 5, 5)
    } else {
        stop("Invalid 'progression' argument")
    }
    ((sum(sal * yrs) * (1 - tax) - child * childcost - house - interest - proptax) / (sum(yrs) + retiredyrs) - travel - car) / 12
}
progressions <- c('average', 'tenure', 'biostats')
life <- round(data.frame(Progressions = progressions,
    NoChild = sapply(progressions, life_style, child = 0),
    OneChild = sapply(progressions, life_style),
    TwoChild = sapply(progressions, life_style, child = 2),
    row.names = 1
) - 1370, 0)
knitr::kable(life, format = 'html',
    caption = 'Monthly difference between costs for idealized life and current life style (1370 a month) using different salary progression data. That is, how much you can save per month for other misc expenses.'
)

Feel free to play around with life_style() with numbers you think are appropriate for you.

## Reproducibility info
library('devtools')
options(width = 120)
session_info()

## Session info-----------------------------------------------------------------------------------------------------------

##  setting  value                                             
##  version  R Under development (unstable) (2014-11-01 r66923)
##  system   x86_64, darwin10.8.0                              
##  ui       X11                                               
##  language (EN)                                              
##  collate  en_US.UTF-8                                       
##  tz       America/New_York

## Packages---------------------------------------------------------------------------------------------------------------

##  package       * version  date       source                                 
##  bibtex          0.3.6    2013-07-29 CRAN (R 3.2.0)                         
##  devtools      * 1.6.1    2014-10-07 CRAN (R 3.2.0)                         
##  digest          0.6.4    2013-12-03 CRAN (R 3.2.0)                         
##  evaluate        0.5.5    2014-04-29 CRAN (R 3.2.0)                         
##  formatR         1.0      2014-08-25 CRAN (R 3.2.0)                         
##  httr            0.5      2014-09-02 CRAN (R 3.2.0)                         
##  knitcitations * 1.0.4    2014-11-03 Github (cboettig/knitcitations@508de74)
##  knitr         * 1.7      2014-10-13 CRAN (R 3.2.0)                         
##  lubridate       1.3.3    2013-12-31 CRAN (R 3.2.0)                         
##  memoise         0.2.1    2014-04-22 CRAN (R 3.2.0)                         
##  plyr            1.8.1    2014-02-26 CRAN (R 3.2.0)                         
##  Rcpp            0.11.3   2014-09-29 CRAN (R 3.2.0)                         
##  RCurl           1.95.4.3 2014-07-29 CRAN (R 3.2.0)                         
##  RefManageR      0.8.40   2014-10-29 CRAN (R 3.2.0)                         
##  RJSONIO         1.3.0    2014-07-28 CRAN (R 3.2.0)                         
##  rstudioapi      0.1      2014-03-27 CRAN (R 3.2.0)                         
##  stringr         0.6.2    2012-12-06 CRAN (R 3.2.0)                         
##  XML             3.98.1.1 2013-06-20 CRAN (R 3.2.0)

References

Citations made with knitcitations (Boettiger, 2015).

[1] Table B-2. Average weekly hours and overtime of all employees on private nonfarm payrolls by industry sector, seasonally adjusted . http://www.bls.gov/news.release/empsit.t18.htm. 2015. URL: http://www.bls.gov/news.release/empsit.t18.htm.

[2] Cost of Raising a Child Calculator. http://www.babycenter.com/cost-of-raising-child-calculator. 2015. URL: http://www.babycenter.com/cost-of-raising-child-calculator.

[3] Essay on the hours faculty members work each day @insidehighered. https://www.insidehighered.com:80/advice/2013/04/22/essay-hours-faculty-members-work-each-day. 2013. URL: https://www.insidehighered.com:80/advice/2013/04/22/essay-hours-faculty-members-work-each-day.

[4] Gross Pay Calculator. http://www.bankrate.com/calculators/tax-planning/quick-tax-rate-calculator.aspx. 2015. URL: http://www.bankrate.com/calculators/tax-planning/quick-tax-rate-calculator.aspx.

[5] Sleep Duration and Its Health Impacts: The Infographic https://sleep.report/sleep-duration-infographic/. 2015. URL: https://sleep.report/sleep-duration-infographic/.

[6] News from The Associated Press. http://hosted.ap.org/dynamic/stories/A/AS_WORKAHOLIC_JAPAN. 2015. URL: http://hosted.ap.org/dynamic/stories/A/AS_WORKAHOLIC_JAPAN.

[7] NIH Postdoc Training Stipends. http://www.nationalpostdoc.org/policy/briefing-room/191-nih-stipend-freeze. 2015. URL: http://www.nationalpostdoc.org/policy/briefing-room/191-nih-stipend-freeze.

[8] Opportunity cost - Wikipedia, the free encyclopedia. http://en.wikipedia.org/wiki/Opportunity_cost. 2015. URL: http://en.wikipedia.org/wiki/Opportunity_cost.

[9] pipredictor.com. http://www.pipredictor.com/. 2015. URL: http://www.pipredictor.com/.

[10] Spending quality family time together is very important. http://msue.anr.msu.edu/news/spending_quality_family_time_together_is_very_important. 2015. URL: http://msue.anr.msu.edu/news/spending_quality_family_time_together_is_very_important.

[11] State taxes: Massachusetts | Bankrate.com . http://www.bankrate.com/finance/taxes/state-taxes-massachusetts.aspx. 2015. URL: http://www.bankrate.com/finance/taxes/state-taxes-massachusetts.aspx.

[12] There is life after academia. http://www.nature.com/news/there-is-life-after-academia-1.15808. 2014. URL: http://www.nature.com/news/there-is-life-after-academia-1.15808.

[13] View Poll. http://www.babycenter.com/4_how-much-time-do-you-spend-with-your-kids_1527797.bc. 2015. URL: http://www.babycenter.com/viewPollResults.htm?pollId=1527797.

[14] We should only work 25 hours a week, argues professor. http://sciencenordic.com/we-should-only-work-25-hours-week-argues-professor. 2015. URL: http://sciencenordic.com/we-should-only-work-25-hours-week-argues-professor.

[15] Whether you're a working mom, stay at home mom, or if your child is a baby or in school - how much time do you typically spend with your child in a day? http://www.circleofmoms.com/question/how-much-time-do-you-spend-your-child-day-1701015. 2015. URL: http://www.circleofmoms.com/question/how-much-time-do-you-spend-your-child-day-1701015.

[16] @DynamicEcology. You do not need to work 80 hours a week to succeed in academia. https://dynamicecology.wordpress.com/2014/02/04/you-do-not-need-to-work-80-hours-a-week-to-succeed-in-academia/. 2014. URL: https://dynamicecology.wordpress.com/2014/02/04/you-do-not-need-to-work-80-hours-a-week-to-succeed-in-academia/.

[17] @jenamcgregor. The average work week is now 47 hours. http://www.washingtonpost.com/blogs/on-leadership/wp/2014/09/02/the-average-work-week-is-now-47-hours/. 2014. URL: http://www.washingtonpost.com/blogs/on-leadership/wp/2014/09/02/the-average-work-week-is-now-47-hours/.

[18] C. Boettiger. knitcitations: Citations for knitr markdown files. R package version 1.0.4. 2015. URL: https://github.com/cboettig/knitcitations.

[19] N. S. Foundation. nsf.gov - NCSES The End of Mandatory Retirement for Doctoral Scientists and Engineers in Postsecondary Institutions: Retirement Patterns 10 Years Later - US National Science Foundation (NSF). http://www.nsf.gov/statistics/infbrief/nsf11302/. 2015. URL: http://www.nsf.gov/statistics/infbrief/nsf11302/.

[20] J. Gould. Are long working hours inevitable for postdocs? : Naturejobs Blog. http://blogs.nature.com/naturejobs/2011/04/01/are-long-working-hours-inevitable-for-postdocs. 2011. URL: http://blogs.nature.com/naturejobs/2011/04/01/are-long-working-hours-inevitable-for-postdocs.

[21] J. Gould. Ask the expert: Can research ever be a 9-5 job? : Naturejobs Blog. http://blogs.nature.com/naturejobs/2014/12/18/ask-the-expert-can-research-ever-be-a-9-5-job. 2014. URL: http://blogs.nature.com/naturejobs/2014/12/18/ask-the-expert-can-research-ever-be-a-9-5-job.

[22] J. Gould. Science blogging : Naturejobs Blog. http://blogs.nature.com/naturejobs/2014/09/18/science-blogging. 2014. URL: http://blogs.nature.com/naturejobs/2014/09/18/science-blogging.

[23] J. Gould. Scientific blogging: Why it might just be good for your CV. http://blogs.nature.com/naturejobs/2015/01/30/scientific-blogging-why-it-might-just-be-good-for-your-cv. 2015. URL: http://blogs.nature.com/naturejobs/2015/01/30/scientific-blogging-why-it-might-just-be-good-for-your-cv.

[24] J. Gould. The postdoc decision : Naturejobs Blog. http://blogs.nature.com/naturejobs/2014/10/03/the-postdoc-decision. 2014. URL: http://blogs.nature.com/naturejobs/2014/10/03/the-postdoc-decision.

[25] A. Hall. The Key To Happiness At Work That Has Nothing To Do With Your Actual Job. http://www.huffingtonpost.com/2015/02/04/happiness-at-work_n_6613358.html. 2015. URL: http://www.huffingtonpost.com/2015/02/04/happiness-at-work_n_6613358.html?ncid=fcbklnkushpmg00000063.

[26] U. T. Larry Copeland. The cost of owning your car? \$\\textasciitilde9,000 a year. http://www.usatoday.com/story/news/nation/2013/04/16/aaa-car-ownership-costs/2070397/. 2013. URL: http://www.usatoday.com/story/news/nation/2013/04/16/aaa-car-ownership-costs/2070397/.

[27] B. McBride. Calculated Risk: New Home Prices: New Record for Average and Median in 2013. http://www.calculatedriskblog.com/2014/01/new-home-prices-new-record-for-average.html. 2014. URL: http://www.calculatedriskblog.com/2014/01/new-home-prices-new-record-for-average.html.

[28] megan. Academic Salary Survey | Amstat News. http://magazine.amstat.org/blog/2014/01/01/academic-salary-survey-2/. 2014. URL: http://magazine.amstat.org/blog/2014/01/01/academic-salary-survey-2/.

[29] OACT. Retirement \& Survivors Benefits: Life Expectancy Calculator. http://www.socialsecurity.gov/cgi-bin/longevity.cgi. 2015. URL: http://www.socialsecurity.gov/cgi-bin/longevity.cgi.

[30] Unknown. Unknown. http://www.mortgagecalculator.org/. Accessed 2015-02-12. 2015. URL: http://www.mortgagecalculator.org/.

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

Use hidden advanced arguments for user-friendly functions

Thu, 11 Dec 2014 00:00:00 +0000

As a user

Imagine that you are starting to learn how to use a specific R package, lets call it foo. You will look at the vignette (if there is one), use help(package = foo), or look at the reference manual (for example, devtools’ ref man). Eventually, you will open the help page for the function(s) you are interested in using.

?function_I_want_to_use

In many packages, there is a main use case that is addressed by the package. A common strategy is to export a main function. That function will likely have a long list of arguments. So as a new user, you are suddenly exposed to a complicated help page and you will want to figure out which arguments you need to use.

As a developer

From the developer’s side, you want to give users control over several details. Each detail you want the user to control involves one more argument in your function. Sooner rather than later, you will have a long list of arguments. This increases the learning curve for new users of your package, and can potentially scare them away. That is contradictory of another goal you have as a developer: you want to get people to use your package.

Lets say that you are developing the function use_me(). If the details you want the users to control are actually arguments of other functions used inside use_me(), then you can simplify your function by using the ... argument. This argument is very well explained at The three-dots construct in R (Burns, 2013). It is very useful and can greatly simplify your life as a developer. Plus, it reduces the length of your help pages, thus making your package more user friendly.

However, if some of the details in use_me() are not arguments to other functions, then the common strategy is to write two functions. One is a low level function with arguments for all the details which might or might not export. Then, you write a second function that is a wrapper for the low level function and pre-specifies values for all the details. See the next minimal example:

# Don't export this function
.use_me <- function(arg1, arg2, verbose = TRUE) {
    if(verbose) message(paste(Sys.time(), 'working'))
    pmax(arg1, arg2)
}

#' @export
use_me <- function(arg1, ...) {
    .use_me(arg1, 0, ...)
}

## Lets see it in action
use_me(1:3)

## 2014-12-11 17:03:32 working

## [1] 1 2 3

use_me(-1:1, verbose = FALSE)

## [1] 0 0 1

In this example, the help page for use_me() is fairly short and friendly. You don’t expect users to be interested in changing arg2 much. Surely you could make it so the non-exported function .use_me() sets a default value for arg2.

Another strategy is to specify inside use_me() the default values for all the arguments you want to use while keeping the list of visible arguments short. That is, maintain the user friendliness of your functions while also giving them control over all the details. That is what you can do using dots() from dots (Collado-Torres, 2014). dots() is a very simple function that checks if ... has a specific argument, and if absent, it returns a default value. It can be seen in action below:

library('dots')
use_me_dots <- function(arg1, ...) {
    ## Default hidden arguments
    arg2 <- dots(name = 'arg2', value = 0, ...)
    verbose <- dots('verbose', TRUE, ...)
    
    ## Regular code
    if(verbose) message(paste(Sys.time(), 'working'))
    pmax(arg1, arg2)
}
use_me_dots(1:3)

## 2014-12-11 17:03:32 working

## [1] 1 2 3

use_me_dots(-1:1, verbose = FALSE)

## [1] 0 0 1

use_me_dots(-1:1, verbose = FALSE, arg2 = 5)

## [1] 5 5 5

dots is my solution to the problem of keeping functions user friendly while giving them control over all the details. The idea is that experienced users will be able to find what the advanced arguments are. While they could find them from the code itself, I do recommend describing the advanced arguments in a vignette targeted for these users.

Complications

Now, while ... is great, you might run into problems when use_me() calls two functions that have different arguments and that don’t have the ... argument. Such a scenario is illustrated below.

status <- function(arg3, status = TRUE) {
    if(status) print(arg3)
    return(invisible(NULL))
}
use_me_again <- function(arg1, ...) {
    res <- .use_me(arg1, 0, ...)
    status(res, ...)
    return(res)
}

## Seems to work
x <- use_me_again(1)

## 2014-12-11 17:03:32 working

## [1] 1

## But nope, it doesn't
use_me_again(1, verbose = FALSE, status = FALSE)

## Error in .use_me(arg1, 0, ...): unused argument (status = FALSE)

This scenario can happen when you are using functions from other packages. It’s happened to me in cases where the main function does have a ... argument but uses several internal functions that don’t use it.

In such situations, you might want to use formal_call() from dots. It figures out which are the arguments formally used by the function of interest and drops out un-used arguments from ..., thus avoiding this type of problem.

use_me_fixed <- function(arg1, ...) {
    res <- formalCall(.use_me, arg1 = arg1, arg2 = 0, ...)
    formal_call(status, arg3 = res, ...)
    return(res)
}

## Works now!
use_me_fixed(1, verbose = FALSE, status = FALSE)

## [1] 1

For a more complicated example, see the dots complex example in the vignette.

Conclusions

As a developer, it is possible to keep your functions user friendly while giving experienced users the option to control the fine tuning arguments which you don’t expect most users will want to tweak. My solution to this problem is implemented in dots (check it’s vignette). I’d love to hear what you think about it! I am specially interested on what users think about the idea of hidden advanced arguments (documented in an advanced users vignette).

I might try to get dots into a repository: probably in Bioconductor since most of the dots code was first implemented for derfinder.

PS I just found a similar function to dots(). It’s berryFunctions::owa() and you can find its code here.

References

Citations made with knitcitations (Boettiger, 2014).

[1] C. Boettiger. knitcitations: Citations for knitr markdown files. R package version 1.0.4. 2014. URL: https://github.com/cboettig/knitcitations.

[2] P. Burns. The three-dots construct in R - Burns Statistics. http://www.burns-stat.com/the-three-dots-construct-in-r/. 2013. URL: http://www.burns-stat.com/the-three-dots-construct-in-r/.

[3] L. Collado-Torres. dots: Simplifying function calls. R package version 1.0.0. 2014. URL: https://github.com/lcolladotor/dots.

Reproducibility

## Reproducibility info
library('devtools')
options(width = 120)
session_info()

## Session info-----------------------------------------------------------------------------------------------------------

##  setting  value                                             
##  version  R Under development (unstable) (2014-11-01 r66923)
##  system   x86_64, darwin10.8.0                              
##  ui       X11                                               
##  language (EN)                                              
##  collate  en_US.UTF-8                                       
##  tz       America/New_York

## Packages---------------------------------------------------------------------------------------------------------------

##  package       * version  date       source                                 
##  bibtex          0.3.6    2013-07-29 CRAN (R 3.2.0)                         
##  devtools      * 1.6.1    2014-10-07 CRAN (R 3.2.0)                         
##  digest          0.6.4    2013-12-03 CRAN (R 3.2.0)                         
##  dots          * 1.0.0    2014-11-14 Github (lcolladotor/dots@a933540)      
##  evaluate        0.5.5    2014-04-29 CRAN (R 3.2.0)                         
##  formatR         1.0      2014-08-25 CRAN (R 3.2.0)                         
##  httr            0.5      2014-09-02 CRAN (R 3.2.0)                         
##  knitcitations * 1.0.4    2014-11-03 Github (cboettig/knitcitations@508de74)
##  knitr         * 1.7      2014-10-13 CRAN (R 3.2.0)                         
##  lubridate       1.3.3    2013-12-31 CRAN (R 3.2.0)                         
##  memoise         0.2.1    2014-04-22 CRAN (R 3.2.0)                         
##  plyr            1.8.1    2014-02-26 CRAN (R 3.2.0)                         
##  RColorBrewer  * 1.0.5    2011-06-17 CRAN (R 3.2.0)                         
##  Rcpp            0.11.3   2014-09-29 CRAN (R 3.2.0)                         
##  RCurl           1.95.4.3 2014-07-29 CRAN (R 3.2.0)                         
##  RefManageR      0.8.40   2014-10-29 CRAN (R 3.2.0)                         
##  RJSONIO         1.3.0    2014-07-28 CRAN (R 3.2.0)                         
##  rstudioapi      0.1      2014-03-27 CRAN (R 3.2.0)                         
##  stringr         0.6.2    2012-12-06 CRAN (R 3.2.0)                         
##  XML             3.98.1.1 2013-06-20 CRAN (R 3.2.0)

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

I wrote dots a month ago and the post itself today during our bi-weekly blog meeting.

An xpd-tion into R plot margins

Fri, 21 Nov 2014 00:00:00 +0000

This is a guest post by Prasad Patil that answers the question: how to put a shape in the margin of an R plot?

The help page for R’s par() function is a somewhat impenetrable list of abbreviations that allow you to manipulate anything and everything in the plotting device. You may have used this function in the past to create an array of plots (using mfrow or mfcol) or to set margins (mar or mai).

Way down toward the end of the list is the often-overlooked xpd parameter. This value specifies where in the plotting device an object can actually be plotted. The default is xpd = FALSE, which means that plotting is clipped, or restricted, to the plotting region. In other words, if your plot has xlim = c(0, 10) and ylim = c(0, 10) and you try to plot the point (-1, -1), it will not appear anywhere in the device.

xpd takes two other values, TRUE and NA, which limit plotting to the figure and device region, respectively. If you’re fuzzy on plotting terms, this tutorial presents those topics well.

Plotting outside the plot

If you want to plot outside of the plotting region, I find that setting xpd = NA easiest since it opens up all external space. We also need to make sure that we keep space outside of the plot so that we have room to place our objects. Let’s say we want to put an ugly border above and below our plot:

# Set xpd=NA and expand the top and bottom margins
par(xpd = NA, mar = par()$mar + c(2.5, 0, 1, 0))
plot(1:10)
# Note that the rectangle we make here has corner coordinates outside of
# our plotting device
rect(-5, 11, 12, 14, col="red")
# Random dots in our rectangluar region
points(runif(100, -4.2, 12.8), runif(100, 11.2, 13.6), col = "green", pch = 19, cex = 1.2)
# And another rectangle for below
rect(-5, -1.7, 12, -3.5, col="red")
points(runif(100, -4.2, 12.8), runif(100, -3.3, -1.8), col = "green", pch = 19, cex = 1.2)

Here we mentally extend the axes of our plot to determine where to put our margin elements. One can imagine a diagonal for the top rectangle running from (-5,11) to (12,14). Neither of these points appear in the plot itself, but we used the established axes to estimate them and plot outside the plotting region.

Images outside the plot

Now let’s say we want to add a logo or other external image in the margin of our plot. We will use R’s png library to load a PNG image and rasterImage() to plot it:

## If needed: install.packages("png")
library(png)
img <- readPNG("logo.png")
par(xpd = NA, mar=par()$mar + c(3, 0, 0, 0))
plot(1:10)
rasterImage(img, 0.5, -2.5, 10.5, -1)

Here we used the png library and the r asterImage() command to read in and plot the “logo.png” file. Based on the previously-known dimensions of the logo, we can choose which points to use as endpoints for the image. Note that this image may appear stretched or contorted depending on the size of your R plot device, and it will not stay consistent if you resize.

Where do I start using Bioconductor?

Thu, 16 Oct 2014 00:00:00 +0000

I was recently asked where do I get started with Bioconductor? and thought this would be a good short post.

What is BioC?

Briefly, Bioconductor (Gentleman, Carey, Bates, and others, 2004) is an open source project that hosts a wide range of tools for analyzing biological data with R (R Core Team, 2014). These analysis tools are bundled into packages which are designed to answer specific questions or to provide key infrastructure. If this sounds like something you are interested in, visit bioconductor.org.

Obviously, you need to know the basics about R in order to use Bioconductor.

Getting started

bioconductor.org has a section in it’s front page titled get started with Bioconductor. There you will find links that explain how to install it or to explore the available packages.

You have a use case

If you have a particular use case in mind, I recommend browsing the software packages and searching for some key words. For example, you might be interested in high throughput sequencing of RNAs and if you search RNAseq or RNA-seq you can find a good set of packages to start. Alternatively, use the biocViews tree menu to explore specific categories of packages.

Once you find a set of packages that have descriptions that appeal to you, explore their vignettes. These are either PDF or HTML documents that explain what the package does to new users. They also exemplify how to tie together the different functions in the package, which is a key piece of information. For example, in the RNA-seq example you will find the DEXSeq package. DEXseq (Anders, Reyes, and Huber, 2012) has a vignette called Analyzing RNA-seq data for differential exon usage with the “DEXSeq” package and from the page of the package you can access the PDF vignette.

Then it’s just a matter of exploring other packages, checking the vignettes and learning as you go.

You don’t have a use case

If you don’t have a specific use case in mind, it might pay off to start by exploring the Bioconductor workflows. These documents explain how to use different packages to accomplish specific type of analyses. They are great to learn what you can do with Bioconductor!

Another option is to look at the previous courses. For example, under the 2008 courses you’ll find to the course R/Bioconductor Curso Intensivo (Spanish) which I taught back in the day. As much as I would like to self promote myself, the best starting point is the most recent BioC20XX course: BioC2014. It has slides showcasing some of the newest packages and tutorials on how to use them.

An alternative is to look at some of the Bioconductor publications which includes books about Bioconductor and research papers describing some of the packages.

Once you find a set of packages that catch your eye, go look at their vignettes just like I explained in the you have a use case scenario.

Help tips

It’s not a matter of whether you will need help learning how to use Bioconductor. It’s just a matter of when. So don’t feel bad about having to ask for help!!

The very first place to start is to look at bioconductor.org at the Help section in the bottom. For example, you can find youtube videos contributed under the community section. There you can also find links to other blog posts explaining how to use Bioconductor. Take a peak at the other sections under Help before using the Bioconductor support site: it’s where you can ask very specific questions and interact with the maintainers of the packages you are using.

Finally, if you are interested in new developments, then check the latest newsletter, for example the October 2014 one.

Good luck using Bioconductor!

References

Citations made with knitcitations (Boettiger, 2014).

[1] S. Anders, A. Reyes and W. Huber. “Detecting differential usage of exons from RNA-seq data.” In: Genome Research 22 (2012), p. 4025. DOI: 10.1101/gr.133744.111.

[2] C. Boettiger. knitcitations: Citations for knitr markdown files. R package version 1.0.2. 2014. URL: https://github.com/cboettig/knitcitations.

[3] R. C. Gentleman, V. J. Carey, D. M. Bates and others. “Bioconductor: Open software development for computational biology and bioinformatics”. In: Genome Biology 5 (2004), p. R80. URL: http://genomebiology.com/2004/5/10/R80.

[4] R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. Vienna, Austria, 2014. URL: http://www.R-project.org/.

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

End of the summer blogger challenge

Fri, 03 Oct 2014 00:00:00 +0000

This summer a few of us at Bmore Biostats agreed to a summer iron blogger challenge. We either had to blog every week or every two weeks. The penalty of not publishing a post in our chosen timeframe was to donate 5 or 10 USD (respectively) to our charity of choice.

I liked the idea of the blogging challenge and thought it would help me keep motivated to keep my blog active during the summer. However, I totally failed and only posted twice during the 16 week summer. That is, I have to donate 14 * 5 = 70 USD. I think that most of us blogged a lot less than what we were hoping to and potentially owe a charity some money. But well, there is no real obligation to donate and I don’t expect others to do so. However, I want to do it. Doing so will help me take a iron blogger challenge more seriously next time.

Kiva

As a charity of my choice, I ended up choosing Kiva. Through them you can loan 25 USD to an individual or group that needs the money to improve their business, home, etc. And in general, you get your money back. Alternatively, you can donate your money from the get go or keep re-loaning it instead of withdrawing it back to your bank.

It is my first time doing something like this, so I am going to ask for the money back and then re-loan it as I see fit. So, because strictly speaking I am not donating it, I decided to increase my entry to 100 USD. That is, 4 Kiva loans.

Kiva recommends lending to different individuals, Kiva partners, and countries to diversify your risk. Being born in Mexico, I wanted to specifically help people from there. If you restrict the search to Mexico, currently there are 13 open loans. From them, I chose:

Aurea administered by Eblock International. She wants to install a floor in her room. It seems like she has used other Kiva loans to slowly build her house.
Sonia administered by Kubo Financiero. She wants to buy recycling materials for her business.
Mujeres Mazahuas Group who are re-stocking their grocery business. The loan is administered by VisionFund Mexico who have gotten loans for over 6 million since 2009. That’s a large chunk of the 16 million that Kiva has sent to Mexico.
Gabino administered by Sistema Biobolsa. He wants a biodigester which will help him in his agriculture business.

Hopefully the loans will be used properly! But like my friend who has experience in micro-banks in Mexico tells me, it is very hard to know that. We’ll see what happens.

For the meantime, here is my receipt.

Join Kiva

For whatever’s worth, here is my Kiva invitation link: kiva.org/invitedby/fellgernon.

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

Concerns that can deter potential orders for developing Shiny apps

Tue, 10 Jun 2014 00:00:00 +0000

A few weeks ago I was invited to a meeting where a group was interested in exploring options for replacing their contract with a propriety software. They invited me because they saw some resemblances between a Shiny application I made and the features they need. It is a relatively small project and it seemed feasible to implement, but well, some details could have been tricky to code. During the meeting I explained what Shiny is, showcased some of the Shiny apps I’ve made, and proposed some options including a simple site password.

A few days after the meeting, they raised three concerns.

Privacy and security of their data. Specially with distribution of the site password.
Technical support. They wanted something more than community support.
Continuity of the project. Specially since their team might lack the technical expertise to support and modify the Shiny app after it is built.

These are all valid concerns and they are not something I have dealt with or been concerned with other Shiny apps I have deployed. Thus, I ended up finding more information and writing up a long reply which I am modifying into a post format here. If you identify other ways to approach these concerns that I missed, share the knowledge please!

Shiny overview

Shiny is an R package that allows creating web applications with R. A user opens the app on their browser, specifies a given set of inputs, these are transmitted to an R session, some code is evaluated with the input options, output is produced and transmitted back to the browser. Because it is so easy to use and useful, it has been very popular. That is the gist of it!

Deploying a Shiny app

Once you develop a Shiny app, you have to deploy it on a server so users can access it through their browsers. Here I describe several options to do so.

The application itself needs “Shiny Server” to run. An implementation of Shiny Server is free via ShinyApps (option 1) or if you have your own server you can install the open source version of Shiny Server (option 2). Alternatively, you can pay for Shiny Server Pro and also install it in your own server (option 3). Note that glimmer.rstudio.com is previous version of ShinyApps and you can no longer get accounts on this server.

Option 1 has the advantage of being completely free and that there is no need to pay for your own server. It does rely on ShinyApps being online, which should not be a main problem since it is hosted on the cloud. Meaning that it is supposed to be robust.

Option 2 has the advantage of having technical support for the server (not the app) from whoever is providing the server (could be a company or the school). However, you need to have someone capable of installing Shiny Server and updating it whenever it’s necessary to do so.

Option 3 allows you to have high level password security (via SSL) to any Shiny app you host. Plus RStudio provides technical support for Shiny. The server technical support would still have to come from whoever is providing the server. The main disadvantage is that this option is very expensive (even with the academic pricing discount).

Here is some information you might be interested on checking about Shiny:

Concerns

Privacy

Privacy issues in terms of password sharing can be limited by changing the password frequently. Privacy in terms of protection versus hacking is a different subject and I could implement something like this demo (username: withr, password: 12345678) as described in this blog post. However, if you want further protection vs hacking, you would need Shiny Server Pro. Here is a demo of the security provided by Shiny Server Pro (with multiple users too). I understand that hacking was not the main concern you had, but well, as I said password-sharing can be mitigated by frequent changes to the app site.

Technical support

I mostly answered the different options regarding technical support when I described the three options for deploying a Shiny app. As for getting support from IT, they would need to learn how to use Shiny.

Continuity

Shiny is one of the most important projects for RStudio so they invest a lot in it, the community provides great answers very quickly, and a lot of people are learning how to use it. If you need an R programmer later on instead of a student, you could describe the job at R Users and employ someone that way. As far as the Biostats department is concerned, I know that several students are using Shiny and some have even published Shiny Apps in the academic literature. Here are some examples:

interAdapt is a Shiny app written by Aaron Fisher (Biostats student) and published here.
shinymethyl, a shiny application developed by Jean-Philippe Fortin described here with a demo here.
committees is a Shiny app made by Alyssa Frazee for verifying your school oral and/or defense committees. It’s described in this post.
ENAR 2014 is a Shiny app made by John Muschelli for the ENAR2014 conference which allowed you to quickly identify talks you might be interested in. It’s described in this post.

Branding

Regarding branding, a Shiny App can be embedded on a website. For illustrating this, I embedded two apps I made here.

Conclusions

In the end, this group decided to use something else instead of a Shiny app for now. I do not know exactly why they made that decision, but I can see that their concerns are probably shared with other potential Shiny users. These concerns are a challenge RStudio must be dealing with and I am curious to know how they would have approached them.

Want more?

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

This post is part of the summer 2014 Bmore Biostats’ iron blogger challenge.

Simple mortgage calculator

Tue, 22 Apr 2014 00:00:00 +0000

Have you ever thought of borrowing some money? A common case is when you have to borrow money for buying a house, which is called a mortgage. Wikipedia (‘Mortgage loan’ entry) goes into much more detail about the definition than what I’ll cover.

One of the aspects you have to evaluate when considering a loan such as a mortgage is how much interest you will be charged and how long the loan will be. Those two determine your monthly payment.

Comparing loans

That is, given the loan amount, you can compare different loans that offer different rates and/or loan lengths. There are many tools out there that allow you to do so and are generally called mortgage calculators. Although the more detailed term is amortization calculator. From wikipedia(‘Amortization calculator’ entry) we find the so called annuity formula which helps you calculate your monthly payment.

Several such mortgage calculators have been features in R Bloggers before such as (C, 2010) and (BioStatMatt). Plus there are many other alternatives on the web.

My use case

However, a month or so ago I got curious about comparing some loans which had a different twist than normal. Given some circumstances, I wanted to compare some loans where the loaner is willing to receive payments every few months yet with interests being compounded (that is, calculated) on a different time frame. Obviously, the easiest would be to compound interest at the same frequency as payments are made. Plus, probably most banks would not offer such flexible loans.

Digging around, I did find modifications to the annuity formula that allow non-monthly payments (Compound Interest Formula) and (What is compound interest?).

shiny app

Being interested in R code, shiny (RStudio Inc.) apps and trying out integrating rCharts (Vaidyanathan, 2013) with shiny, I ended up coding my own mortgage calculator which you can find here (Simple Mortage Calculator). The code is publicly available at GitHub.

It has a couple of simple inputs:

Amount to borrow
Interest rate
Loan duration
Payment frequency
How frequently interests are compounded
Month of the first payment

The option to begin the loan now but accept the first payment much later is there also because of the complicated use case I had in mind. Again, that is a feature a bank will most likely not offer. But it’s something I needed to take into consideration.

Going back to the app, I tried explaining all the inputs as much as possible. The output is relatively straight forward.

First, there is a line with focus D3 interactive plot (Line Chart with View Finder) which shows the principal (the amount you owe) over months as well as how much you’ve paid already. The bottom panel allows you to zoom into a specific time range as shown below.

Next, the information is shown in more detail as an interactive data table (DataTables) 12 months at a time. The interactive part makes it very easy to search for a specific month and look at the state of the loan.

Finally, you can download the amortization table in CSV format for your records.

PS You can also access the app via shinyapps at this url. It is the first time I’ve deployed an app there as I wanted to test shinyapps out.

Conclusions

The experience of coding the shiny app (Simple Mortage Calculator) was interesting as I did learn a couple of new things. The same was true for figuring out the calculations for the more flexible options.

Finally, but not least, the shiny app was useful for my use case and was informative for comparing different loan options. Hopefully it will be useful for other users!

References

Citations made with knitcitations (Boettiger, 2014).

Mortgage loan - Wikipedia, the free encyclopedia. Wikipedia http://en.wikipedia.org/wiki/Mortgage_loan
Amortization calculator - Wikipedia, the free encyclopedia. Wikipedia http://en.wikipedia.org/wiki/Amortization_calculator
C, (2010) Mortgage Calculator (and Amortization Charts) with R. R-Chart http://www.r-chart.com/2010/11/mortgage-calculator-and-amortization.html
BioStatMatt, Mortgage Refinance Calculator | BioStatMatt. BioStatMatt http://biostatmatt.com/archives/1908
Compound Interest Calculator. http://www.calculator.net/compound-interest-calculator.html
Compound Interest. http://math.about.com/library/weekly/aa042002a.htm
Simple Mortgage Calculator. http://glimmer.rstudio.com/lcolladotor/mortgage/
Novus Partners, Line Chart With View Finder - NVD3. http://nvd3.org//examples/lineWithFocus.html
DataTables (table plug-in for jQuery). DataTables (jQuery plug-in) https://datatables.net/
Carl Boettiger, (2014) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations
Ramnath Vaidyanathan, (2013) rCharts: Interactive Charts using Polycharts.js.
RStudio , Inc. , (2014) shiny: Web Application Framework for R. http://CRAN.R-project.org/package=shiny

Reproducibility

sessionInfo()

## R version 3.1.0 (2014-04-10)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitcitations_0.5-0 bibtex_0.3-6        knitr_1.5.27       
## 
## loaded via a namespace (and not attached):
##  [1] codetools_0.2-8 digest_0.6.4    evaluate_0.5.3  formatR_0.10   
##  [5] httr_0.3        RCurl_1.95-4.1  stringr_0.6.2   tools_3.1.0    
##  [9] XML_3.98-1.1    xtable_1.7-3

Check other @jhubiostat student blogs at Bmore Biostats as well as topics on #rstats.

rMaps Mexico map

Wed, 26 Feb 2014 00:00:00 +0000

It’s exciting when great people help each other get things done

This is a simple networking story, which might not be surprising to some but I was happily surprised by it. This is how the story goes:

Two weeks ago rMaps (Vaidyanathan, 2014) was released. After making a blog post about it I thought about using it to make a map of the homicide rate in Mexico over the recent years. First, I had the question of how to make custom maps with rMaps. @tyokota had the same question and started asking Ramnath about it in rMaps issue 6. Then I realized I needed a specific file with the map information. Google lead me to @diegovalle who has created the map from official Mexican sources, downloaded the homicide data, cleaned it, and made several maps and analyses: all his work is very impressive! I thought that it’d be very cool if @diegovalle and Ramnath connected, and they did! I saw them interacting via Twitter (here and here) and via GitHub. After sharing @diegovalle’s work with my friends, it turned out that some old friends already knew him (here and high school friends). Another friend was interested in additional features and I suggested her to contact @diegovalle via Twitter: he quickly replied as you can see here.

Beyond how impressive rMaps and @diegovalle’s work on mexican data are, I was amazed by the willingness to help each other and how great people easily connected. I believe this is one of the great features of both GitHub and Twitter where you can share your code, ask questions, try to answer them, meet people working with your tools, etc. You can even offer to PayPal a beer like @tyokota did.

After all their great work, now someone like me (aka, without knowing javascript, Datamaps, etc) can walk you through an example of making an interactive choropleth map showing the homicides rates in Mexico from 1997 to 2013.

Homicides rates in Mexico 1997-2013

The first thing we need to make a custom map using rMaps is a topojson file which in this case specifies the mexican states boundaries. This process is explained in more detail by @tyokota at custom-map which you can view here.

In this particular example, INEGI which is the National Institute of Statistics and Geography of Mexico has a map of the mexican states. @diegovalle explained how to download it here.

But before doing so, you might to install topojson like I did below following the installation instructions. In the terminal:


## Install node.js following instructions at https://github.com/mbostock/topojson/wiki/Installation
brew install node
## Install topojson
npm install -g topojson

## Download map info from INEGI (Mexican official source)
curl -o estados.zip http://mapserver.inegi.org.mx/MGN/mge2010v5_0.zip
## Decompress file
unzip estados.zip
## Create shapefile
ogr2ogr states.shp Entidades_2010_5.shp -t_srs "+proj=longlat +ellps=WGS84 +no_defs +towgs84=0,0,0"
## id-property needed so that DataMaps knows how to color the map
topojson -o mx_states.json -s 1e-7 -q 1e5 states.shp -p state_code=+CVE_ENT,name=NOM_ENT --id-property NOM_ENT

Now that we have the topojson file mx_states.json we need to get the actual homicide data. @diegovalle has gone through the whole process of acquiring the data from official mexican sources and cleaning it. Lets download it.

# Download crime data
## From crimenmexico.diegovalle.net/en/csv
## All local crimes at the state level
download.file("http://crimenmexico.diegovalle.net/en/csv/fuero-comun-estados.csv.gz", 
	"fuero-comun-estados.csv.gz")

The data is not completely ready for us to use it and we need to reshape it a bit. In particular, we want to consider only the intentional homicides and group the data by state and date. We can get this to work by using dplyr (Wickham & Francois, 2014).

## Load required packages
library("dplyr")

## Load the crime data
crime <- read.csv("fuero-comun-estados.csv.gz")

## Only intentional homicides
crime <- subset(crime, crime == "HOMICIDIOS" & type == "DOLOSOS")

## Sum homicides by firearm, etc and group by state and date
hom <- crime %.%
  filter(year %in% 1997:2013) %.%
  group_by(state_code, year, type) %.%
  summarise(total = sum(count, na.rm = TRUE),
            population = mean(population) ) %.%
  mutate(rate = total / population * 10^5) %.%
  arrange(state_code, year)

## How are states coded?
summary(hom$state_code)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    1.00    8.75   16.50   16.50   24.20   32.00

We have the slight inconvenience that states are coded as integers from 1 to 32 instead of using their names. Using another of the files supplied by @diegovalle we can merge the codes. This requires using the foreign (R Core Team) package for loading a dbf file and then merging both data sets with plyr (Wickham, 2011).

## Needed for read.dbf
library("foreign")

## The dbf from the state shapefile needed to merge state_code with state
## names
codes <- read.dbf("states.dbf")
codes$NOM_ENT <- iconv(codes$NOM_ENT, "windows-1252", "utf-8")
codes$CVE_ENT <- as.numeric(codes$CVE_ENT)
codes$OID <- NULL
names(codes) <- c("state_code", "name")

## Load plyr for join(). Loading it before creates a problem with the dplyr
## call above
library("plyr")

## Names needed for the map
hom <- join(hom, codes)

## Lets look at the data
head(hom)

##   state_code year    type total population   rate           name
## 1          1 1997 DOLOSOS   355     958126 37.051 Aguascalientes
## 2          1 1998 DOLOSOS    66     975585  6.765 Aguascalientes
## 3          1 1999 DOLOSOS    27     992515  2.720 Aguascalientes
## 4          1 2000 DOLOSOS    14    1009215  1.387 Aguascalientes
## 5          1 2001 DOLOSOS    22    1026437  2.143 Aguascalientes
## 6          1 2002 DOLOSOS    26    1044578  2.489 Aguascalientes

Great! We now have state names under name and the intentional homicide rate under rate (in homicides per 100,000) for each specific year. We can thus proceed to making the interactive choropleth map using the ichoropleth function described by Ramnath here. This requires specifying the topojson file which is specified via dataUrl, the name of the map specified via scope and the most tricky part (for me at least) is that we need to specify the setProjection. These are all properties of the Datamaps library. In particular, the wiki describes how to use custom maps but this requires some javascript knowledge.

## Make the map
library("rMaps")
d1 <- ichoropleth(rate ~ name, data = hom, ncuts = 9, pal = 'YlOrRd', 
    animate = 'year',  map = 'states'
)
## Note that I am hosting the mx_states.json in Dropbox
## but if you are doing it locally, you only need
## dataUrl = "mx_states.json"
d1$set(
  geographyConfig = list(
   dataUrl = "https://dl.dropboxusercontent.com/u/10794332/mx_states.json"
  ),
 scope = 'states',
 setProjection = '#! function( element, options ) {
   var projection, path;
   projection = d3.geo.mercator()
    .center([-89, 21]).scale(element.offsetWidth)
    .translate([element.offsetWidth / 2, element.offsetHeight / 2]);

   path = d3.geo.path().projection( projection );
   return {path: path, projection: projection};
  } !#'
)
d1$save('rMaps.html', cdn = TRUE)

The end result is shown below:

You can also share the map using the publish method as shown below:

d1$publish("Intentional homicides rates for Mexico 1997-2013")
## You'll need a GitHub account

You will get a link to the rCharts viewer such as this one or if you prefer, you can also view the result using Pagist as shown here.

The code presented in this post was written by @diegovalle which can you view here and Ramnath which is shown here. I also figured out the trick of hosting the topojson file at Dropbox from @tyokota’s code as I was running into Access-Control-Allow-Origin errors when hosting it in my academic website. Finally, but not least, Ramnath appropriately insists that all of this would not be possible without libraries such as Datamaps.

References

Citations made with knitcitations (Boettiger, 2014).

Hadley Wickham, Romain Francois, (2014) dplyr: dplyr: a grammar of data manipulation. http://CRAN.R-project.org/package=dplyr
R Core Team , (2014) foreign: Read Data Stored by Minitab, S, SAS, SPSS, Stata, Systat, Weka, dBase, …. http://CRAN.R-project.org/package=foreign
Carl Boettiger, (2014) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations
Hadley Wickham, (2011) The Split-Apply-Combine Strategy for Data Analysis. Journal of Statistical Software 40 (1) 1-29 http://www.jstatsoft.org/v40/i01/
Ramnath Vaidyanathan, (2014) rMaps: Interactive Maps from R.

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] rMaps_0.1.1         plyr_1.8            foreign_0.8-59     
## [4] dplyr_0.1.1         knitcitations_0.5-0 bibtex_0.3-6       
## [7] knitr_1.5          
## 
## loaded via a namespace (and not attached):
##  [1] assertthat_0.1     digest_0.6.4       evaluate_0.5.1    
##  [4] formatR_0.10       grid_3.0.2         httr_0.2          
##  [7] lattice_0.20-24    rCharts_0.4.2      RColorBrewer_1.0-5
## [10] Rcpp_0.11.0        RCurl_1.95-4.1     RJSONIO_1.0-3     
## [13] stringr_0.6.2      tools_3.0.2        whisker_0.3-2     
## [16] XML_3.95-0.2       xtable_1.7-1       yaml_2.1.10

Check other topics on #rstats.

Automatically coloring your R output in the terminal using colorout

Mon, 17 Feb 2014 00:00:00 +0000

Thanks to Alyssa Frazee I just learned about the colorout package (Aquino, 2013). It modifies R so that the output is in different colors, making it much more pleasant to use R in the terminal.

Do note that colorout is not available from CRAN, but you can easily install by following the instructions on the colorout site (Official site) reproduced below:

download.file("http://www.lepem.ufc.br/jaa/colorout_1.0-2.tar.gz", destfile = "colorout_1.0-2.tar.gz")
install.packages("colorout_1.0-2.tar.gz", type = "source", repos = NULL)

The next step is to then load colorout automatically when I start R. The problem is that I don’t use R solely on the terminal. I easily figured out how to do so thanks to the error message you get when attempting to load colorout on the R GUI. I thus ended up adding the following lines to my .Rprofile (both locally and in the cluster):

## Change colors when running R in the terminal
if (Sys.getenv("TERM") == "xterm-256color") library("colorout")

Now I have pretty R output in the terminal! Thanks again Alyssa! See her original tweet below:

problem: R output in Terminal isn't colorful. SOLUTION: http://t.co/Vd6OoRoUU5
— Alyssa Frazee (@acfrazee) February 17, 2014

colorout has been around for a while and was even at CRAN for some time. I guess that I’m just late to the party.

If the default colorout colors don’t work for you, check functions such as setOutputColors256. This post shows how you can do it and includes screenshots of the output. Other package details are included here and here.

References

Citations made with knitcitations (Boettiger, 2014).

Jakson Aquino, (2013) colorout: Colorize R output on terminal emulators. http://www.lepem.ufc.br/jaa/colorout.html
colorout. http://www.lepem.ufc.br/jaa/colorout.html
Carl Boettiger, (2014) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitcitations_0.5-0 bibtex_0.3-6        knitr_1.5          
## 
## loaded via a namespace (and not attached):
##  [1] codetools_0.2-8 digest_0.6.4    evaluate_0.5.1  formatR_0.10   
##  [5] httr_0.2        RCurl_1.95-4.1  stringr_0.6.2   tools_3.0.2    
##  [9] XML_3.95-0.2    xtable_1.7-1

Check other topics on #rstats.

rMaps released

Mon, 10 Feb 2014 00:00:00 +0000

Ramnath Vaidyanathan just released his new R interactive package, rMaps (Vaidyanathan, 2014). The packages relies on the development version of his widely known rCharts package (Vaidyanathan, 2013) as well as javascript libraries that specialize in maps. If you don’t know Ramnath, he is one of the most active R developers out there!! You can see that from his GitHub profile.

The package is very new and still under development, but I bet that Ramnath released it to get us users excited and maybe find some helpful hands to document it and further develop it.

I don’t know about you, but I surely got excited about the package from his intro video:

It’s a simple screen cast with good music.

In the GitHub rMaps repository you can find the simple installation instructions as well as three different examples. They all work if you run them in the latest version of RStudio otherwise you might run into a couple minor hiccups like I did.

Just to get you excited, this is the third example where you can easily add markers with pop ups.

suppressMessages(library("rMaps"))

map <- Leaflet$new()
map$setView(c(51.505, -0.09), zoom = 13)
map$tileLayer(provider = "Stamen.Watercolor")
map$marker(c(51.5, -0.09), bindPopup = "Hi. I am a popup")
map

You can view the interactive version of the example here – I’m sure that a feature will be added later to make it easy to share the maps you make.

Overall, I think that this is a great start and I look forward to using it. For now, don’t be discouraged with the lack of documentation. I’m sure that if you ask nicely Ramnath will answer asap!

References

Citations made with knitcitations (Boettiger, 2014).

Carl Boettiger, (2014) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations
Ramnath Vaidyanathan, (2013) rCharts: Interactive Charts using Polycharts.js.
Ramnath Vaidyanathan, (2014) rMaps: Interactive Maps from R.

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] rMaps_0.1           knitcitations_0.5-0 bibtex_0.3-6       
## [4] knitr_1.5          
## 
## loaded via a namespace (and not attached):
##  [1] codetools_0.2-8    digest_0.6.4       evaluate_0.5.1    
##  [4] formatR_0.10       grid_3.0.2         httr_0.2          
##  [7] lattice_0.20-24    plyr_1.8           rCharts_0.4.2     
## [10] RColorBrewer_1.0-5 RCurl_1.95-4.1     RJSONIO_1.0-3     
## [13] stringr_0.6.2      tools_3.0.2        whisker_0.3-2     
## [16] XML_3.95-0.2       xtable_1.7-1       yaml_2.1.10

Check other topics on #rstats.

How to upload files to Dropbox and Google Docs from R

Wed, 05 Feb 2014 00:00:00 +0000

Have you ever wondered whether you can upload files from R to Dropbox and/or Google Docs? I recently asked myself this question while making my most recent Shiny app (more later).

The answer is yes, you can upload files from R to these cloud services!

Dropbox

As far as I know, the best R package for uploading files to Dropbox is rDrop (Ram & Temple Lang, 2012). The whole setup is very well explained in it’s GitHub repository (Karthik). Basically you have to:

Install the package and it’s dependencies (some are only on GitHub).
Create a Dropbox app in your account.
Get the credentials info.
Authorize your access to the Dropbox app from R.
Save that info for later use.

dropbox_save()

Then uploading any type of R object to Dropbox becomes as easy as using the dropbox_save() function.

For example, lets save a simple vector with random data.

suppressMessages(library("rDrop"))

## Define credentials or load them if you defined them already
## dropbox_credentials <- dropbox_auth('Your app key', 'Your app secret')
load("dropbox_credentials.Rdata")

## Lets create some random data
set.seed(20140205)
x <- rnorm(1000)

## Lets check the args of the uploading function
args(dropbox_save)

## function (cred, ..., list = character(), file = stop("'file' must be specified"), 
##     envir = parent.frame(), precheck = TRUE, verbose = FALSE, 
##     curl = getCurlHandle(), ext = ".rda", .opts = list()) 
## NULL


## Then lets upload it to dropbox on the public folder
done <- dropbox_save(dropbox_credentials, x, file = "public/x")

## The result has some information, like the path of where you upload the
## file
names(done)

##  [1] "revision"     "rev"          "thumb_exists" "bytes"       
##  [5] "modified"     "client_mtime" "path"         "is_dir"      
##  [9] "icon"         "root"         "mime_type"    "size"

done$path

## [1] "/public/x.rda"

You can now actually download the “x.rda” file from here. That’s in case that you also wanted to share the file, and is obviously optional.

Note that you can get the link from withing R and don’t even need to use the Dropbox site.

dropbox_share(dropbox_credentials, file = "public/x.rda")

## $url
## [1] "https://db.tt/xzf3huXf"
## 
## $expires
## [1] "Tue, 01 Jan 2030 00:00:00 +0000"

dropbox_put()

What if you want to upload an actual file and not only R objects? That’s where dropbox_put() shines. Below is an example where we create an image, save it as a pdf file, and upload it to Dropbox.

## Lets create a sample file, in this case a pdf file
pdf("dropboxFig.pdf")
hist(x, freq = FALSE, col = "light blue")
tmp <- dev.off()

## Lets check the args for the uploading function
args(dropbox_put)

## function (cred, what, filename = what, curl = getCurlHandle(), 
##     ..., verbose = FALSE, contentType = "application/octet-stream") 
## NULL


## Now, lets upload the file
done <- dropbox_put(dropbox_credentials, what = "dropboxFig.pdf", filename = "public/dropboxFig.pdf")

## Again, the result contains some information about the file
names(done)

##  [1] "revision"     "rev"          "thumb_exists" "bytes"       
##  [5] "modified"     "client_mtime" "path"         "is_dir"      
##  [9] "icon"         "root"         "mime_type"    "size"

You can view the result here.

Google Docs

From what I found, it seems to me that RGoogleDocs (Temple Lang) is the package you’ll want to use for interacting with Google Docs from R. The manual (Temple Lang) explains all what you pretty much need to know. You should know though that you can only upload certain types of files.

For example, you can upload a text file as shown below.

suppressMessages(library("RGoogleDocs"))

## Load password
load("gpasswd.Rdata")

## Something to write
text <- "Hello world!\n"

## Authentificate
auth <- getGoogleAuth("lcolladotor@gmail.com", gpasswd)

## Connect to Google
con <- getGoogleDocsConnection(auth)

## Check the args for the uploading function
args(uploadDoc)

## function (content, con, name, type = as.character(findType(content)), 
##     binary = FALSE, asText = FALSE, folder = NULL, replace = TRUE, 
##     ...) 
## NULL


## Upload the file
done <- uploadDoc(content = text, con = con, name = "testFile.txt", type = "txt")

You can view the file here.

Conclusions

rDrop (Ram & Temple Lang, 2012) is very cool and easy to use. Compared to Google Docs, you have much greater flexibility on the type of files you can upload. I guess that will change in the future if there is a Google Drive from R package.

References

Citations made with knitcitations (Boettiger, 2014).

karthik, karthik/rDrop. GitHub https://github.com/karthik/rDrop
An simple R interface to Google Documents. http://www.omegahat.org/RGoogleDocs/run.html
Carl Boettiger, (2014) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations
Karthik Ram, Duncan Temple Lang, (2012) rDrop: Dropbox R interface.. http://github.com/karthikram/rDrop
Duncan Lang, RGoogleDocs: Primitive interface to Google Documents from R.

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] RGoogleDocs_0.7-0   rDrop_0.3-0         stringr_0.6.2      
##  [4] plyr_1.8            ROAuth_0.92.0       digest_0.6.4       
##  [7] RCurl_1.95-4.1      bitops_1.0-6        RJSONIO_1.0-3      
## [10] knitcitations_0.5-0 bibtex_0.3-6        knitr_1.5          
## 
## loaded via a namespace (and not attached):
## [1] evaluate_0.5.1 formatR_0.10   httr_0.2       tools_3.0.2   
## [5] XML_3.95-0.2   xtable_1.7-1

Check other topics on #rstats.

Login to the cluster, request a node and change to your project directory in a single command

Wed, 11 Dec 2013 00:00:00 +0000

To be able to do RNA-seq research work in large multi-sample studies you have to be able to analyze large files and thus frequently use a powerful computing environment. In my case, this means that I have to login to a computing cluster frequently. This is a common task for other biostatisticians (like those that do brain imagining studies) and many other people. When I am working on a project, I generally have to login to the cluster and then change the directory to the location where I have my project files.

A local cluster is normally composed of a login machine (enigma2 in my case) from where you can request to work on a node. There are several options for controlling this process of requesting a node, and in our institution we use a Sun/Oracle Grid Engine. When a cluster has a lot of users, you want to dedicate the login machine as much as possible to handling login requests and assigning nodes for people to use. This means that you want to minimize doing any other kind of operations on the login machine, such as input/output operations.

Basic workflow

This means that the steps I normally follow before getting to work on my project are:


## Open the terminal
## Login to enigma
ssh username@enigma2.etc.edu

## Request a node to work on interactively
qrsh

## Change to the directory where I have my project files
cd projectDir

## Done!

For a long time I have used an alias for the login step. I have this alias in my local .bashrc file:


## In local .bashrc file
alias enigma="ssh username@enigma2.etc.edu"

Thus saving a tiny bit of typing:


## Open terminal
## Login to enigma
enigma
## Etc
qrsh
cd projectDir

When the projectDir gets complicated, I make an alias on my .bashrc file on the cluster. For example:


## In cluster .bashrc file
alias pdir="cd /very/complicated/path/to/projectDir/"

And finally, I can accomplish the setup task with minimal typing:


## Open terminal
enigma
qrsh
pdir

Using ssh config

This section was added after Kasper Hansen’s comment.

You can edit the ~.ssh/config file (check how to set it up, explained differently and the manual) to make things even better. This is how mine looks like:


Host enigma
	User username
    Hostname enigma2.etc.edu
	ForwardX11 yes

I like the ssh -X (or -Y) option so I can later view plots in X11 when running R. That is why the ForwardX11 option is present.

Then you can use the following command to ssh into the cluster.


ssh enigma

Or if you prefer, simplify the bash alias to:


## In local .bashrc file
alias enigma="ssh enigma"

The sections below have been edited to assume that you are configured the enigma host shortcut in your ~.ssh/config file.

Needed a new strategy

The previous strategy works and I had been very comfortable with it. However, at times you might forget to request a node from the cluster to do your work interactively. This is specially true for me when I only plan on using a few git commands. But when many users forget this, it becomes a problem and our cluster manager had to send us a reminder:

*ALWAYS* work on a cluster node rather than on enigma2.
Enigma2 is a *single machine* with many people trying to use it to gain access to the cluster.
Even for tar commands, cp commands, wc commands ... first qrsh to a node.

Profiles in iTerm2

Just a few days after we got this reminder, I decided to take a look at the iTerm2 profile menu. There are plenty of options for customizing your terminal, but the ones I mainly ended up using are:

Working directory -> Directory : -> choose a directory in my laptop. Generally the location of my git repository for version controlling the code of a given project.
Command -> Send text at start -> an alias from my local .bashrc file as shown in the image below (the alias is qr).

The first case above is nice, but the real power comes from the second case. Since I can pretty much evaluate any command, I asked myself if I could set up a profile that automatically logs in to the cluster? Can it also request a node interactively? And even go to my project directory?

cluster `qr` alias

I then remembered that Samuel Younkin explained to us how to set up qrsh to automatically change the directory to the directory from which you invoked qrsh (Younkin, 2013). I modified things a little bit and saved this qrsh version as the qr alias on the cluster:


## In cluster .bashrc file

## change dir automatically when using qrsh
## Details: https://github.com/rkostadi/BiocHopkins/wiki/Useless-Tips-&-Code-Snippets
if [ -f ~/.bash_pwd ]; then
    source ~/.bash_pwd
    rm ~/.bash_pwd
fi
alias qr='echo "cd $PWD" > ~/.bash_pwd; history -w; qrsh'

Note that I tried using the qrsh -ac option, but couldn’t get to pass a variable. Doing so in theory would remove the need to create the .bash_pwd temporary file.

ssh and change dir in one command

Then googling I found how to ssh and change directory in one command (Frosty, 2009):


ssh -t enigma 'cd /very/complicated/path/to/projectDir/; bash'

The problem I soon encountered was that I couldn’t qrsh right after because the command was not been found. Some setup files are not been read even after using bash -l like shown here:


ssh -t enigma 'cd /very/complicated/path/to/projectDir/; bash -l'

local `qr` alias

With our cluster administrator’s help, I was finally able to find how to do all of this in a single command:


## Requires the code by Sam Younkin to work (or the version I modified)
ssh -t enigma 'cd /very/complicated/path/to/projectDir/; source /etc/profile; echo "cd $PWD" > ~/.bash_pwd; history -w; qrsh'

Finally, I created the qr alias in my local machine. This alias:

performs the ssh connection to the login machine of our cluster (enigma2),
then changes the directory to the project directory,
loads the necessary setup files so qrsh can be found on the path,
creates the temporary .bash_pwd file (you could even make this more concise by echoing the directory of interest to the temp file),
saves the command history as recommended by Samuel Younkin,
requests an interactive node (with your default settings) by executing qrsh.


## In local .bashrc file

# qrsh
alias qr="ssh -t enigma 'cd /very/complicated/path/to/projectDir/; source /etc/profile; echo \"cd \$PWD\" > ~/.bash_pwd; qrsh'"

Note the use of the backslash to delay the execution of $PWD. I want it to be executed on the cluster, not on my local machine.

Glory!

So now using iTerm2 I can simply use the shortcut for my cluster profile which runs the local qr alias and the next thing I have is a terminal with an interactive session on the cluster and located in my project directory. Sweet! =)

Plus I also have the cluster qr alias for doing what Samuel Younkin previously described.

Can do it better?

If you have suggestions on how to improve this, let me know!

Extra aliases

The following two aliases take your local current directory basename and use it for accessing /very/complicated/path/to/projectDir/basename. This is useful if you use an organization similar to mine:

Projects dir
- Project 1 dir
- Project 2 dir

The full paths are different in my computer and in the cluster, but once you are in /very/complicated/path/to/projectDir/ it is all the same on both locations.

The first one runs qrsh while the second one doesn’t request a node for interactive work.


## In local .bashrc file

## qrsh-basename
alias qs='LEODIR=`basename $PWD`; ssh -t enigma "cd /very/complicated/path/to/projectDir/$LEODIR/; source /etc/profile; echo \"cd \$PWD\" > ~/.bash_pwd; history -w; qrsh"'
## basename, but no qrsh
alias qq='LEODIR=`basename $PWD`; ssh -t enigma "cd /very/complicated/path/to/projectDir/$LEODIR/; source /etc/profile; echo \"cd \$PWD\" > ~/.bash_pwd; history -w; bash"'

References

Citations made with knitcitations (Boettiger, 2013).

Frosty, (2009) How can I ssh directly to a particular directory?. How can I ssh directly to a particular directory? - Stack Overflow http://stackoverflow.com/questions/626533/how-can-i-ssh-directly-to-a-particular-directory
BiocHopkins. GitHub http://bit.ly/19zLZeD
Carl Boettiger, (2013) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations

Recap

If you got lost, these are the basic modifications you need to make to your local and cluster .bashrc files.

Check the history of this post here.

Creating awesome reports for multiple audiences using knitrBootstrap

Tue, 10 Dec 2013 00:00:00 +0000

As a biostatistics student, I use R very frequently when analyzing data. At the same time, I interact with other researchers, some who know how to use R (R crowd) and some who don’t (yet!): no-R crowd. This means that I have to be able to communicate my results to two crowds. It is important that I can quickly provide the code in case that the R savvy want to look at it: maybe they find a bug and report it ^^. Ideally I want to avoid having to write (organize, share, etc) two crowd-specific reports.

A solution to this problem is to create reproducible reports that contain the R code, the results, and interpretation. For the specific scenario I am talking about, reproducibility is a plus, however I believe that it is important for research; albeit not the topic of this post. One of the strongest packages out there to create such reports is knitr (Xie, 2013). It is specially easy to create Rmd files from which you can generate HTML reports. Then using RStudio you can share them via RPubs, a private folder on Dropbox, etc. From example, this is a presentation without the slide formatting I shared more than a year ago.

Using knitr (Xie, 2013) is definitely a step in the right direction. However, you soon find yourself desiring a better template. This is where knitrBootstrap (Hester, 2013) comes in. This package was initialized in March 20th, 2013 by Jim Hester and hosted on it’s GitHub repository. I was sold on the idea early on and I am now making this post in part as a tribute to celebrate that it has been available via CRAN for nearly 5 months now.

So what can you do with knitrBootstrap (Hester, 2013)? In my opinion, you get the ideal solution (or very close at the least) to the problem I described at the beginning. Basically, you get a HTML report that has the interpretation and results which is what the no-R crowd wants to read, and the R code easily available at the click of a button for the R crowd. In addition, the report is much more nicely formatted which is pleasant to the eye. Furthermore, a menu with the sections is included which is very useful when navigating the report and for jumping to specific sections. To save space, the plots are saved as thumbnails and you can click on them to get the full view. Finally, you can choose to display toggle menus for allowing the users to change the default text and code formatting.

How do you use this package? The main workhorse is the knit_boostrap() function. The initial arguments are similar to those you find in knitr::knit() while the new features are controlled using:

boot_style You can select out of 11 or so options for the default formatting. Basically, you choose one of the Bootstrap themes available.
code_style Similar to boot_style but for controlling the appearance of the code chunks.
chooser Allows you to control if you want a toggle menu so the user can choose (hence the name) the bootstrap and/or code styles.
thumbsize For controlling the size of figure thumbnails.
show_code Whether by default the code is shown. I set this to FALSE in order to get a report that by default is accessible for the no-R crowd. The R crowd can then click to see the code for each code chunk or use the menu on the bottom to show all the code at once.
show_output Similar to show_code but for controlling the visibility of the output produced from the code. I set this to TRUE as you normally want to show the output to both the no-R and R crowds.
show_figure Whether you want to show the plots or not.
graphics Used only for controlling the toggle menus for the bootstrap and code styles.

Once you have decided which options you want to use, it is as simple as running the following code for your Rmd file (named file.Rmd in the example):

## Install if needed
install.packages("knitrBootstrap")

## knit with knitrBootstrap
library("knitrBootstrap")
knit_bootstrap("file.Rmd", code_style = "Brown Paper", chooser = c("boot", "code"), 
    show_code = FALSE)

Things get a tiny bit more complicated if you want to use RStudio. You basically have to modify your .Rprofile file, then load RStudio and change the settings to weave files with knitrinstead of using Sweave. Then, you have to use knitr::render_html() on the Rmd file itself. Below is a short example of the .Rprofile modified to use knitrBootstrap and the basic Rmd example.

You can view the final output here. Note that you might need to click on “hide toolbars” (a RPubs option) to clearly view the menus on the bottom.

If you are like me and use Textmate as your text editor, you can knit the Rmd files with knitrBootstrap and preview them directly on the Textmate viewr using a command like this (modified from the SWeave bundle):

Other usage options are described in the knitrBootstrap help page.

To close off, let me emphasize how useful it is to be able to generate a single report that is pleasant to the eye, contains all the information, and is easily sharable for both the R and no-R crowds. Plus it’s reproducible!

I really like this package and would like to thank Jim Hester for this great package! Keep up the good work! I even use knitrBootstrap in derfinderReport which generates reports on the results from derfinder, a package that I am currently developing.

References

Citations made with knitcitations (Boettiger, 2013).

Carl Boettiger, (2013) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations
Yihui Xie, (2013) Dynamic Documents with {R} and knitr. http://yihui.name/knitr/
Jim Hester, (2013) knitrBootstrap: Knitr Bootstrap framework.. http://CRAN.R-project.org/package=knitrBootstrap

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitrBootstrap_0.9.0 knitcitations_0.4-7  bibtex_0.3-6        
## [4] knitr_1.5           
## 
## loaded via a namespace (and not attached):
##  [1] codetools_0.2-8 digest_0.6.4    evaluate_0.5.1  formatR_0.10   
##  [5] httr_0.2        markdown_0.6.3  RCurl_1.95-4.1  stringr_0.6.2  
##  [9] tools_3.0.2     XML_3.95-0.2    xtable_1.7-1

Check other topics on #rstats.

Third Student Cultural Mixer

Wed, 20 Nov 2013 00:00:00 +0000

Note: the Fall event will be student and post doc only while the Spring event will be open to the whole department. This is to smooth the integration process for the new students. A survey showed that at least some first year students would be more likely to either attend or present if it’s a student-only event.

Trying to reduce the memory overhead when using mclapply

Thu, 14 Nov 2013 00:00:00 +0000

I am currently trying to understand how to reduce the memory used by mclapply. This function is rather complicated and others have explained the differences versus parLapply (A_Skelton73, 2013; lockedoff, 2012 ) and also made it clear that in mclapply each job does not know if the others are running out of memory and thus cannot trigger gc (Urbanek, 2012).

While I still struggle to understand all the details of mclapply, I can successfully use it to reduce computation time at the expense of a very high memory load. I am still looking for tips on how to reduce this memory load.

Here is what I have done.

Problem setting

I have a large data set on the form of a data.frame. I want to apply a function that works using subsets of the data.frame without the need for communication between the chunks, and I want to apply the function fast. In other words, I can safely split the matrix and speed the computation process using mclapply.

While this works, I would like to minimize memory consumption.

Toy data

Here is just some toy data for the example.

## The real data set is much larger than this
set.seed(20131113)
data <- data.frame(matrix(rnorm(1e+05), ncol = 10))
dim(data)

## [1] 10000    10

Approach 1

The first approach I have used is to pre-split the data and then use mclapply over the split data. For illustrative purposes, lets say that the function I want to apply is just rowMeans.

## Pre-split the data
dataSplit <- split(data, rep(1:10, each = 1000))

## Approach 1
library("parallel")
res1 <- mclapply(dataSplit, rowMeans, mc.cores = 2)

This gets the job done, but because my real dataSplit is much larger in memory, using say 8-10 cores blows up the memory.

Best way to pre-split?

If I know that if I am using $ n $ number of cores (in this example $ n=2 $ ) and the data set has $ m $ rows, then one option for approach #1 is to split the data into $ n $ chunks each of size $ m / n $ (rounding if needed).

## Pre-split the data into m/n chunks
dataSplit1b <- split(data, rep(1:2, each = 5000))

## Approach 1b
res1b <- mclapply(dataSplit1b, rowMeans, mc.cores = 2)

The memory needed is then in part determined by the chunksize (1000 vs 5000 shown above). One excellent suggestion (via Ben) is to reduce the memory load using this approach is to just smaller chunks. However, the runtime of the function I want to apply (rowMeans in the example) is not very sensible to the chunksize used, thus using very small chunks is not ideal as it increases computation time. Finding the sweet point is tricky, but using chunksizes of $m / (2n) $ could certainly help memory wise without majorly affecting computation time.

Approach 2

One suggestion (via Roger) is to use an environment in order to minimize copying (and thus memory load) while using mclapply over a set of indexes.

## Save the split data in an environment
my.env <- new.env()
my.env$data1 <- dataSplit1b[[1]]
my.env$data2 <- dataSplit1b[[2]]

## Function that takes indexes, then extracts the data from the environment
applyMyFun <- function(idx, env) {
    eval(parse(text = paste0("result <- env$", ls(env)[idx])))
    rowMeans(result)
}

## Approach 2
index <- 1:2
names(index) <- 1:2
res2 <- mclapply(index, applyMyFun, env = my.env, mc.cores = 2)

## Same result?
identical(res1b, res2)

## [1] TRUE

Approach 3

Another suggestion (via Roger) is to save the data chunks and load them individually inside the function that I pass to mclapply. This does not seem ideal in terms of having to create the temporary chunk data files. But I would expect this method to have the lowest memory footprint.

## Save the chunks
for (i in names(dataSplit1b)) {
    chunk <- dataSplit1b[[i]]
    output <- paste0("chunk", i, ".Rdata")
    save(chunk, file = output)
}

## Function that loads the chunk
applyMyFun2 <- function(idx) {
    load(paste0("chunk", idx, ".Rdata"))
    rowMeans(chunk)
}

## Approach 3
res3 <- mclapply(index, applyMyFun2, mc.cores = 2)

## Same result?
identical(res1b, res3)

## [1] TRUE

Computation time comparison

Computation time wise, approaches 2 and 3 do not seem very different. Approach 1b seems a tiny bit faster. [Edit: the order of the best approach might change slightly if you re-run this code]

library("microbenchmark")
micro <- microbenchmark(mclapply(dataSplit1b, rowMeans, mc.cores = 2), mclapply(index, 
    applyMyFun, env = my.env, mc.cores = 2), mclapply(index, applyMyFun2, mc.cores = 2))
micro

## Unit: milliseconds
##                                                     expr   min    lq
##            mclapply(dataSplit1b, rowMeans, mc.cores = 2) 17.43 19.97
##  mclapply(index, applyMyFun, env = my.env, mc.cores = 2) 17.05 19.20
##               mclapply(index, applyMyFun2, mc.cores = 2) 17.19 23.11
##  median    uq   max neval
##   21.41 26.00 65.53   100
##   20.60 23.92 43.67   100
##   24.56 28.39 46.99   100

library("ggplot2")
autoplot(micro)

Memory wise comparison

Relying on the cluster tools for calculating the maximum memory used, I ran each approach (1b, 2, and 3) ten times each using 2 cores using the scripts available in this gist. The maximum memory used showed no variability (within an approach) and the results are that approach 1b used 1.224G RAM, approach 2 used 1.176G RAM, and approach 3 used 1.177G RAM. Not a huge difference. Due to having to write and then load, approach 3 was slower than the other two.

Re-doing the previous test but using 20 cores lead to very similar wall clock computation times between all three approaches and to approaches 1b and 2 for 2 cores. This is due to the nature of the example, aka rowMeans is fast even with the larger chunks. Approach 1b used 7.728G RAM, approach 2 used 7.674G RAM, and approach 3 used 7.690G RAM. Hm…

Using 20 cores with previously created data files (either the split data for approaches 1b and 2, or the chunk files for approach 3) has a very different memory footprint. Approach 1b used in average 6.0744G RAM, approach 2 used 4.2647G RAM , and approach 3 used 2.6545G RAM.

Edit

Ryan from (Ryan 2013) contributed a fourth approach which used 6.794G RAM when starting from scratch with 20 cores. This approach definitely beats the other ones under the condition of starting from scratch. Note that just creating the data object uses 558.938M RAM: multiplied by 20 it would be around 10.92G RAM.

Conclusions

Using 2 or 20 cores, approach 2 beat by a very small margin approaches 3 and 1b in terms of memory usage. However, all approaches failed in terms of not having the memory blow up as you increase the number of cores when starting from scratch.

If a lower memory option is used for splitting the data and creating the chunk files, approach 3 seems like the winner in terms of memory usage. So in pure terms of lowering the memory load on mclapply approach 3 wins, although you still need to create the chunk files and do so without much memory usage.

If you have any ideas or suggestions, please let me know! Thank you!

References

Citations made with knitcitations (Boettiger, 2013).

A_Skelton73, (2013) understanding the differences between mclapply and parLapply in R. understanding the differences between mclapply and parLapply in R - Stack Overflow http://stackoverflow.com/questions/17196261/understanding-the-differences-between-mclapply-and-parlapply-in-r
lockedoff, (2012) Using mclapply, foreach, or something else in [r] to operate on an object in parallel?. Using mclapply, foreach, or something else in [r] to operate on an object in parallel? - Stack Overflow http://stackoverflow.com/questions/11036702/using-mclapply-foreach-or-something-else-in-r-to-operate-on-an-object-in-par
[R-sig-hpc] mclapply: rm intermediate objects and returning memory . https://mailman.stat.ethz.ch/pipermail/r-sig-hpc/2012-October/001534.html
[Bioc-devel] Trying to reduce the memory overhead when using mclapply . https://stat.ethz.ch/pipermail/bioc-devel/2013-November/004930.html
Carl Boettiger, (2013) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] parallel  stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
## [1] microbenchmark_1.3-0 ggplot2_0.9.3.1      knitcitations_0.4-7 
## [4] bibtex_0.3-6         knitr_1.5           
## 
## loaded via a namespace (and not attached):
##  [1] codetools_0.2-8    colorspace_1.2-4   dichromat_2.0-0   
##  [4] digest_0.6.4       evaluate_0.5.1     formatR_0.10      
##  [7] grid_3.0.2         gtable_0.1.2       httr_0.2          
## [10] labeling_0.2       MASS_7.3-29        munsell_0.4.2     
## [13] plyr_1.8           proto_0.3-10       RColorBrewer_1.0-5
## [16] RCurl_1.95-4.1     reshape2_1.2.2     scales_0.2.3      
## [19] stringr_0.6.2      tools_3.0.2        XML_3.95-0.2      
## [22] xtable_1.7-1

Scripts

The scripts are available in this gist. The main one is testApproach.R while the other ones are just job-submitters.

Check other topics on #rstats.

Creating your Jekyll-Bootstrap powered blog for R blogging

Sat, 09 Nov 2013 00:00:00 +0000

As you might have noticed, I recently decided to move Fellgernon Bit from Tumblr to GitHub. There are a couple of reasons why I made this change.

I wanted a more professional-looking blog.
- There are not many R blogs on Tumblr, and well, long text posts are not really meant for Tumblr.
Better code highlighting.
- I had enabled R code highlighting using the highlighting instructions from Jeffrey Horner (Horner, Part I). The instructions are great! I guess I just got lazy to tweak the CSS of my Tumblr blog to fix some things I didn’t like.
- I also want to be able to highlight code from other languages, like bash.
Make it easier to embed any HTML code.
- Yes, you can write posts at Tumblr in pure HTML but sometimes things break as Tumblr doesn’t like them.
Easier process of writing posts using Rmd files.
- The steps in Horner’s guide (Horner, Part II) are quite involved. Instead of those steps, I was just knitting my posts and then manually copying the html code in the body tags, then pasting it in Tumblr.
Easy way to host figures and link to them.
- I have previously hosted pictures through Picasa Web Albums, but with the push towards Google Plus they made it harder to embed solo pictures (you can embed Google+ posts though).

I had also previously heard about Jekyll-powered blogs, specially thanks to Yihui Xie’s blog (Xie, English version). At the time when I first learned about this type of blog it seemed challenging so I never dedicated the time to really learn about it. What ended up movitivating me to do so was our new student bloggers group @jhubiostat.

Let me describe how I set up Fellgernon Bit @ GitHub.

Setting up Fellgernon Bit @ GitHub

Creating the Jekyll-Bootstrap blog

My main guide to setup the blog comes from a post by J. Fisher (Fisher, 2012) which points to another of his posts (Fisher, 2012b).

Install Jekyll

Before doing anything, I first had to install Jekyll. To do so I recommend checking the installation documentation (Jekyll Installation).

To install Jekyll, I needed to have Ruby and RubyGems. Since I use MacPorts these are the steps I used:


## Update MacPorts, can take a while!!
sudo port selfupdate
sudo port upgrade outdated

## Search for Ruby and install it
## I think that Jekyll needs 1.9+ since I had 1.8 already installed
port search ruby
sudo port install ruby19
## Set as default version
sudo port select --set ruby ruby19

## Check which version you have
ruby --version

## I manually set mine to be ruby1.9
cd /opt/local/bin
sudo ln -s ruby1.9 ruby
ruby --version 
## This is mine:
## ruby 1.9.3p448 (2013-06-27 revision 41675) [x86_64-darwin12]

With Ruby in place, I could then install jekyll.


sudo gem install jekyll

Note that you can also do this after initializing the blog.

Initialize blog

Following the instructions (Fisher, 2012b) I then acquired all the setup files for Jekyll-Bootstrap. The installation instructions are quite straight forward as you will see (JB Quick Start).

First, I created the lcolladotor.github.com repository on GitHub. As my GitHub username is lcolladotor, by default GitHub will consider the lcolladotor.github.com repository as a GitHub pages repository and publish it. Thus creating lcolladotor.github.io/.

Once the repository was created, I added the Jekyll-Bootstrap files.


## Initialize repo with Jekyll-Bootstrap files
git clone https://github.com/plusjade/jekyll-bootstrap.git lcolladotor.github.com
cd lcolladotor.github.com/
git remote set-url origin git@github.com:lcolladotor/lcolladotor.github.com.git
git push -u origin master

Change theme to Twitter-2.0

Jekyll-Bootstrap includes several themes (JB Themes) which you check on their theme explorer. I think that J. Fisher’s blog looks good with Twitter-2.0 and decided to follow his lead. Furthermore, he has customized a few things (check his blog’s history) which I just plan on using.


## Install Twitter-2.0 theme
rake theme:install git="https://github.com/gdagley/theme-twitter-2.0"

Code highlighting

The default Jekyll highlighting setup uses pygments. I am basing my statement on the documentation for the highlight argument in ?knitr::render_jekyll:

which code highlighting engine to use: for pygments, the Liquid syntax is used (default approach Jekyll)

So I installed Pygments.


## Install Pygments, assuming you have Python installed
curl -O http://python-distribute.org/distribute_setup.py
sudo python distribute_setup.py
sudo easy_install Pygments

Note that (Fisher, 2012b) details some other steps. I basically copied his syntax.css file and saved it as /assets/themes/twitter-2.0/css/syntax.css

Note that you do have to add to _includes/themes/twitter-2.0/default.html the following line:

<link href="/assets/themes/twitter-2.0/css/syntax.css" rel="stylesheet" type="text/css">

As you can see on my own file in line 23 (currently) I am using a Liquid syntax instead of specifying the full path.

Import posts

Next I had to import my posts from Tumblr. Jekyll has a varied set of tools that allow you to quickly import posts which I recommend using if you are migrating your blog (Blog Migrations).

The Tumblr importer allows you to either import them into HTML format or Markdown format. I first tried Markdown, but ran into some problems. So I decided to import them into HTML format, and rename them to Markdown.


## Install importing tools
sudo gem install jekyll-import --pre

## Import the posts
cd ~/
mkdir tmp
/opt/local/bin/ruby1.9 -rubygems -e 'require "jekyll/jekyll-import/tumblr"; JekyllImport::Tumblr.process("http://fellgernon.tumblr.com", "html", true, false, false)'
## There is a post that the converter didn't like, so I had to set to "private" mode


## Rename from HTML to Markdown
cd _posts/tumblr/
for i in *html; do echo $i; filename=$(basename "$i"); filename2="${filename%.*}"; echo $filename2; mv $i ${filename2}.md; done

## Move the posts to _posts
mv ~/tmp/_posts/tumblr/*md ~/WhereYouHaveYourLocalGitRepoCopy/lcollado.github.com/_posts/

I then had to do some manual edits on the posts, specially for code chunks. For example, this post has R chunks which I had to surround by:

To complete the posts, I made sure the YAML front matter was correct and defined categories since I only had tags in Tumblr.

Viewing your blog locally

Once you have the posts, you can use Jekyll to view your blog locally.


## Go to your directory
~/WhereYouHaveYourLocalGitRepoCopy/lcollado.github.com/

## Generate the static view of your blog
jekyll serve

In my case, jekyll serve prompted a couple of errors which I had to go to the posts and edit. For example, some HTML tag that I opened but didn’t close. Stuff like that.

Once you run the previous command, you can then view your site on your browser at http://localhost:4000/

Customizing

The very first thing you must change is the _config.yml file. The file you copied from Jekyll-Bootstrap includes instructions on how to do so. If you learn by example, take a peek at mine.

In my case, I use Disqus for comments and Google Analytics for tracking visits. Since I was also using Disqus for my Tumblr blog, I used Disqus’ tools to migrate the comments. Finally, I use FeedBurner for my RSS feed, which I was able to update to the new RSS generated by Fellgernon Bit @ GitHub. Thus, the subscribers to feeds.feedburner.com/FellgernonBit didn’t have to change anything.

I also added AddThis Smart Layers with bit.ly shortening to the site and a ClustrMap as you can see in _layouts/page.html.

As for math, I am currently using the rdiscount flavor of Markdown and loading the MathJax script as you can see in _layouts/default.html. However, note that we shoudn’t ask too much power out of Markdown for math (Xie, 2013).

To use rdiscount locally, I also had to install it. I have also tried out kramdown.


## Markdown flavors
sudo gem install rdiscount
sudo gem install kramdown

## I don't remember why I also installed this:
sudo gem install nokogiri

I then customized the index and added other pages:

index.html
- Used code from Jason Fisher to add the pagination.
about.html
- Simple description
_includes/themes/twitter-2.0/default.html
- I added a CSS drop down menu (constructed with CSS Menu Maker) with the blogs I follow. For this I also had to modify _assets/themes/twitter-2.0/css/style.css.
- I modified the footer to mention that my posts are under the Creative Commons License version 3 BY-NC-SA. An idea I took from Yihui Xie’s blog.

Since I am interested in getting my blog to be a part of R-bloggers, I created an RSS specific for the posts under the rstats category to comply with their requirements (Add your blog!).

For the category-specific RSS I used feed.category.xml (Jekyll RSS Feed Templates) and created rss-rstats.xml [see atom-rstats.xml if you prefer Atom RSS] which is then processed and converted to feeds.feedburner.com/FellgernonBit-rstats.

Note that I use FeedBurner to then create Twitter updates linking to back to the posts. Also, for the rstats one I add the #rstats hashtag to the post.

Writing an R post

J. Fisher again explains in detail how to blog using Knitr and Jekyll (Fisher, 2012). Since I use TextMate2 as my text editor, I went ahead and modified his code and created a command to do so.

Basically, I modified the knit command from the SWeave TextMate bundle and added J. Fisher’s code. I saved it under the Menu Actions of the SWeave bundle and linked it to alt + e.

For this to work, I use Google Chrome with the Markdown Preview extension installed.

This is the code for the TextMate command. You could certainly make it much more concise and drop things, but I like to see a quick preview of the post. However, I still recommend using jekyll serve for a final thorough preview of the post before pushing it to GitHub.

Things I might add

Carl Boettiger has a great site –specially the Lab Notebook part– and he describes all the tools he used for making it (Boettiger, 2013). One of the features I would like to add to my site is the history button on the right sidebar he has on his posts which link to the GitHub history of the post. For example check this post and its history. I think that it can be very useful for those interested in any edits made to the posts. For example, I had to fix a couple of things on one of my recent posts as you can see here.

Looking around, I found the code needed for this to work in Carl’s repository: _includes/sidebar.html, _plugins/jekyll-labnotebook-plugins.

However, GitHub currently does not support Jekyll plugins. Meaning that if you use any plugins, you have to create the site locally and then push it to GitHub instead of just pushing the new posts and letting GitHub handle creating the site. It’s not a huge deal, but for now I’ll rely on GitHub and keep things simple for me.

If I were to do it again

Once you install Jekyll, I think the easiest way to set up your own blog is to fork another person’s blog

J. Fisher’s,
Yihui Xie’s,
Carl Boettiger’s,
mine,
or someone else blog you like.

Then delete their posts, give attribution, and customize things like the blog title and other things you would have to customize on any blogging platform (RSS, comments provider, etc).

Obviously Jekyll gives you much more control over your site, but at the same time, you must also dedicate enough time to learn what to change and make sure that things don’t break.

Post template

To finish describing my setup, here is my post template.

Wrap up

I hope that you will find some of this information useful if you are thinking of starting your R blog powered by Jekyll-Bootstrap and hosted by GitHub.

Note that since the blog is hosted in GitHub, others can send you pull requests to fix things and/or create guest posts. Just as easily you can make it a multi-author blog by changing the collaborators setting of the GitHub repository.

This wouldn’t have been possible without J. Fisher’s posts and many other resources available online.

References

Citations made with knitcitations (Boettiger, 2013b).

@jeffreyhorner, Blog with R Markdown and tumblr: Part I. Jeffrey Horner http://jeffreyhorner.tumblr.com/post/25804518110/blog-with-r-markdown-and-tumblr-part-i
@jeffreyhorner, Blog with R Markdown and tumblr: Part II. Jeffrey Horner http://jeffreyhorner.tumblr.com/post/25943954723/blog-with-r-markdown-and-tumblr-part-ii
Yihui Xie, Statistics, R, Graphics and Fun | Yihui Xie. http://yihui.name/en/
Jason Fisher, (2012) Blog with Knitr and Jekyll. http://jfisher-usgs.github.io/r/2012/07/03/knitr-jekyll/
Jason Fisher, (2012) Jekyll Build on Windows. http://jfisher-usgs.github.io/lessons/2012/05/30/jekyll-build-on-windows/
Installation. Jekyll â¢ Simple, blog-aware, static sites http://jekyllrb.com/docs/installation/
Jekyll Quick Start | ruhoh universal static blog generator. http://jekyllbootstrap.com/usage/jekyll-quick-start.html
Using Themes | ruhoh universal static blog generator. http://jekyllbootstrap.com/usage/jekyll-theming.html
Blog migrations. Jekyll â¢ Simple, blog-aware, static sites http://jekyllrb.com/docs/migrations/
Tal Galili, add your blog!. R-bloggers http://www.r-bloggers.com/add-your-blog/
jekyll-rss-feeds. GitHub https://github.com/snaptortoise/jekyll-rss-feeds
Yihui Xie, (2013) Markdown or LaTeX? | Yihui Xie. http://yihui.name/en/2013/10/markdown-or-latex/
Carl Boettiger, (2013) About this site. Lab Notebook http://carlboettiger.info/README.html
Carl Boettiger, (2013) knitcitations: Citations for knitr markdown files. http://CRAN.R-project.org/package=knitcitations

Reproducibility

sessionInfo()

## R version 3.0.2 (2013-09-25)
## Platform: x86_64-apple-darwin10.8.0 (64-bit)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitcitations_0.4-7 bibtex_0.3-6        knitr_1.5          
## 
## loaded via a namespace (and not attached):
##  [1] codetools_0.2-8 digest_0.6.4    evaluate_0.5.1  formatR_0.10   
##  [5] httr_0.2        RCurl_1.95-4.1  stringr_0.6.2   tools_3.0.2    
##  [9] XML_3.95-0.2    xtable_1.7-1

My strategy for quickly grading homeworks

Fri, 08 Nov 2013 00:00:00 +0000

For the past year and a half I have been a teaching assistant (TA) for the Statistical Methods in Public Health I to IV (140.621 to 140.624) courses. As part of being a TA for these courses, we have to grade between 30 and 50 homeworks every two weeks or so: four problem sets per eight week terms. For example, I now have to grade 37 homeworks:

These courses have been polished over time and the instructors do a great job organizing the TA group. For every problem set that we have to grade, we are given a full answer key that is later published in the course site so all the students can access it. As TAs we also receive general grading comments from the instructor; typically a two page document describing the key points that we should expect the students to be able to answer.

As part of the general grading comments, we are asked to give feedback beyond checkmarks such as “good job”. When the student completely misses the answer, we can simply refer them to the answer key: if they miss too many, we ask them to re-do the homework. Things get more complicated when the student misses part of the question and writing comments specific to that answer can get very time consuming. We can always use “More detailed discussion on results needed. See answer key.” but it can be nice for the student if we give them a hint on what they did wrong.

As a student @jhubiostat we agree to dedicate five hours per week to TA responsibilities. In the courses I have been a TA for, the responsibilities include getting familiar with the content, attending TA office hours (either general questions or STATA questions), attending the TA meetings, grading the exams, and grading the homeworks. It might sound like a lot, but I currently feel like I can it all done within the five hours per week. To do so, I obviously need to be efficient grading the homeworks.

Here is how I do it.

Setup

Programs used:

Adobe Acrobat Pro which I believe we can get through Hopkins for free.
Alfred v2 with Powerpack purchased/installed.

Why Adobe Acrobat?

Adobe Acrobat allows you to post “stamps”. It already includes a checkmark and a red X. I also imported some simple images I made to stamp the homework as: satisfactory, unsatisfactory, or incomplete. We have to clearly mark one of these options at the top of the graded homeworks.
Some of the PDF files we get are large, so I use the Save as -> reduced size pdf tool from Adobe Acrobat to save space and make it easier for the course administrator to upload the graded homeworks to CoursePlus (the tool the school uses for managing courses).

Why Alfred v2?

For grading homeworks all I really need is a software that can remember my recent copy-pastes and make it easy for me to choose among them to paste them back. For example, ClipMenu can do the job.
Alfred v2 has a Clipboard feature which does the above. Compared to ClipMenu it has the advantage of allowing you to search within the recent copy-pastes, making it easy to find the comment you want to paste.

If you are on Windows or Linux, there must be some programs that have this functionality.

Grading workflow

Most used comment: Good job!

Let me begin with a very simple case. Basically, you find a answer that is well done and you want to compliment the student for it.

In the homework I am currently grading, the students have to compute the 95 percent confidence interval for the difference in two proportions using the formula:

$$ \hat p_1 - \hat p_2 \pm 1.96 \times \sqrt{ \frac{ \hat p_1 (1 - \hat p_1)}{ n_1} + \frac{ \hat p_2 (1 - \hat p_2)}{ n_2} } $$

which leads to (0.00142, 0.00798) with the data they are using.

An anonymous student (I tried to choose examples that are not identifiable) wrote an answer that has very similar numbers to those we expect. Since they included the STATA output I can recognize that they used rounded numbers (oddly only for sample 1 but not sample 2) which lead the student to slightly different values. That is not a big deal and I think it deserves the “good job!” comment. So using the “stamps”, I added a checkmark. Then I typed “Good job!” and finally, I copied it so I can paste it the next time I want to add this comment.

The image below shows the state at which I am copying the comment.

Moving on, I later found another student that has the correct answer presented in a different format. If “Good job!” is the most recent comment on my clipboard, I can simply paste it. If it is not, then I can use Alfred v2 to show me my recent “copies” and select “Good job!” from the list. In this case it is under cmd + 2; you can also use the arrows and the return key, or the mouse. Although note that Alfred v2 is designed in such a way that you only have to use the keyboard to gain efficiency.

The end result is a green checkmark with the “Good job!” comment pasted into it. The next time I use cmd + v (the regular paste shortcut in Mac), “Good job!” will be pasted. So you only really need to invoke Alfred v2 when you want to change comments.

Detailed comments: where the strategy pays off!

You obviously do not all this setup just for pasting “Good job!” everywhere. However, when grading homeworks you will encounter much more complicated cases where you might need to write a short explanation, point the student to the answer key, etc. To save time you want to minimize having to type the long comments each time you need to. That is where this strategy pays its dividends.

In this example, I first find a plot that looks very similar to what the students were asked to do. It has a couple of mistakes. First, the age-gender group categories are not labeled in the plot, which makes it impossible to know which group is which unless you know how agegen is coded. Given that it is the first time students were asked to make such a detailed plot, I can expect other students to make the same mistake. So I copy that comment and add it to my clipboard history as shown below.

This particular plot has another problem in that the confidence bands look much smaller than expected. The estimates per se look comparable to the expected plot, so I suspect that the student either used the wrong standard errors or forgot to multiply them by 1.96. So I copy this comment separately and add it to my clipboard history. Maybe other students made the same mistake, maybe they did not.

Having copied both comments separately, I am now ready for cases where they forget to label the X axis, use the wrong confidence bands, or make both mistakes together.

Later on, I find another homework that has labels on the X axis. So instead of typing my comment, I simply invoke Alfred v2 and search the appropriate comment as shown below.

If my comment list is getting long, using Alfred v2 I can easily search my clipboard history by typing keywords. In this particular case, I know that my comment had something to do with the “axis”, so typing it in I quickly find the comment I want to use.

And voila! The comment is pasted!

Alternatives

An alternative implementation is to create a spreadsheet where you manually save the comments you are using while grading the homework. When you want to use them, you open the spreadsheet, select the comment, manually copy it, and then you are set to paste it. Using Alfred v2 basically makes this a bit faster, but the key point is to save the comments in a way that you can re-use them and combine them if necessary.

If you want more pointers on this alternative, ask @YennyWebbV who is an expert at it.

Wrap up

I hope that others find this strategy useful. In my case, I went from taking 9-11 hours to grade homeworks when I first began as a TA to grading them in 3-4 hours.

If you are interested in reading more posts from students @jhubiostat check out bmorebiostat.com.

"Bmore Biostats is born!" by Leonardo Collado Torres at http://t.co/CXVuni60gh pic.twitter.com/uIovRg1bAP
— bmorebiostats (@bmorebiostats) November 8, 2013

Bmore Biostats is born!

Thu, 07 Nov 2013 00:00:00 +0000

In recent weeks, I have met with a group of students @jhubiostat interested in blogging about their research, tutorials, pieces of R code, among plenty of other subjects. Within this group we had the idea to aggregate our blogs so it would be easier for others to follow us and to easily promote our own blogs to a much larger audience. Basically, do what R-bloggers does but focused on blogs from students at Johns Hopkins Biostatistics. Ideally, once a student publishes a new post, our site would pick it up and promote it.

A quick search revealed that making such a site was technically possible via several WordPress plugins available. We initally tried doing so via a wordpress.com account and quickly noticed that we needed a self-hosted wordpress account.

paper.li

In the meantime, we also explored the possibility of creating a paper using paper.li which resulted in the BmoreBiostats paper. The free version allows you to select up to 25 RSS feeds (among other sources) as the source of new content that you either publish

in the morning and afternoon,
daily,
weekly.

This platform offers several options for promoting new content which we implemented. It can create posts to Twitter (see below), Facebook and LinkedIn.

New posts from students @jhubiostat http://t.co/jnHXs7qRPJ
— bmorebiostats (@bmorebiostats) November 7, 2013

You can also embed it on a webpage just like I did in my own website.

The drawbacks are that customization is limited, specially if you do not like how it looks. It also does not provide you with a RSS feed. However, I did manage to get that to work via Facebook as you can notice here.

We have also heard some comments about paper.li been associated with spammers. I agree with this if you set the paper to be published either twice per day or daily. Weekly wise, I do not see it as a spam generator. However, this is in contradiction with the original goal of promoting posts as soon as they are made, and only when there are new posts.

Nevertheless, in my opinion, the paper.li alternative can work well as a weekly summary of the posts. Something like the Sunday data/statistics link roundoups at SimplyStatistics.

planetaki.com

John found this resource and implemented a quick test. It looks very simple, yet it includes full length posts and was very easy to setup. Note that the material there is deleted after 7 days since they assume that you check it more than once per week. We saw this as a potential negative feature, plus there is no native support for social media.

bmorebiostat.com

Amanda and Jean-Philippe [thanks for trying out Wordpress__.com__!] figured out the WordPress solution, got us a hosted service and reserved the domain bmorebiostat.com. Then it was just a matter of choosing a plugin to do the job: we are using FeedWordPress.

We split the work and some tweaked the layout and added nice pictures of the six founding blog authors. Thanks to what I learnt making Fellgernon Bit I added the social features: AddThis Smart Layers, connections to social media, Disqus comments (they will not be used much), and FeedBurner RSS feeds.

I like how the site looks and it fully achieves our initial goal. Yay!

Currently, BmoreBiostats is set up in a way that only the beginning of each post is shown. We can also set it up to contain full posts, but then the R code highlighting needs to be polished out.

Current implementation

The full current workflow is illustrated below:

As a group we are now thinking of dropping the paper.li route. Well, the only option we might use is the weekly one. One strong argument in favor of dropping the paper.li route is that one site avoids any dilution given by having two. Furthermore, we do not want to be seen as spammers although some shameless self-promotion is not so bad either (something I learnt from @hspter).

I guess that I am the only one still in favor of weekly summaries and using the feature of embeding the paper in a website (like here). To minimize the dilution, all the paper.li links point to bmorebiostat.com. You can also argue that there might be some interested only on the weekly summaries. Aka, we are just giving others options!

Maybe I am just reluctant to delete the BmoreBiostats paper so soon after I finally completed it. However, some of the work involved is not going down the drain since the social media accounts needed for the paper are now being used by bmorebiostat.com.

Note that we do not see much of a problem with the fact that bmorebiostat.com involves monthly fees as we are hoping to get some department support (cross fingers!).

Going forward

We are very excited that we have implemented the aggregator of student blogs from Johns Hopkins Biostatistics. We believe that it will be helpful to others including prosprective students. Now that the aggregator is practically finished, we can now move unto writing exciting posts!

If you are a student @jhubiostat –or a former student– and you have a Biostatistics blog that you want to add to bmorebiostat.com, let us know! If you are a current student @jhubiostat and need help getting started, contact us!

Be sure to follow us! Here is a hint why:

It seems clear that @jhubiostat is just dominating all the social media @bmorebiostats
— Simply Statistics (@simplystats) November 8, 2013

Quickly making posters with PosterGenius

Tue, 29 Oct 2013 00:00:00 +0000

It is time to revive Fellgernon Bit from it’s deep hibernation period. A couple of very motivated Ph.D. students from my department (John, Alyssa, Amanda, Jean-Philippe, Elizabeth, etc) are organizing a blogging group. The idea is to review ideas, give suggestions, learn blogging technicalities, write blog posts, review them, and post them. It’s a great idea! Plus it should us keep our blogs active.

So for my first post I am going to talk about PosterGenius. It’s a simple to use piece of software for making posters. In my case, I presented a poster at the 7th Annual Young Investigators Symposium and Poster Session on Genomics and Bioinformatics. It was a relatively small event and I had a limited amount of time to make the poster. The idea behind PosterGenius is that you separate your poster into several sections (intro, methods, results, references), have some material for each (text and or figures), and just have to put it together with a simple background.

Of course, there are plenty of other tools for doing this. But in my case, I was able to make a poster in lunch hour by using slides from a presentation on the same subject and just organizing the content. The basic steps are:

Choose a poster template, number of columns, height and width.
Choose how many sections and their titles.
Fill in the authors, affiliations, title and institutional logos.
Enter the pictures using their picture manager. You might have to choose the appropriate size of the pictures (zoom percent). For example, my slides had white space on the borders, which PosterGenius did not know, so some space was being wasted.
Review it.
Print it.

In the following post you can find some pictures from the creation of the poster in question.

PosterGenius has the cool feature of creating a poster, a presentation, and a handout from the same material. For other meetings, I have actually printed out a couple of handouts to give to those interested in the material.

While it is not free (they have discounts for students), I think that PosterGenius is a very simple to use, produces good looking posters, and their optimal distance to read feature is quite accurate.

LIBD video

Wed, 17 Jul 2013 00:00:00 +0000

Starting this week, I’ll be doing my research with the Lieber Institute for Brain Development (LIBD) as Andrew Jaffe's first Ph.D. student there. My main advisor will continue to be Jeff Leek which is great for me. I’ll have access to massive data sets at Lieber and will face the challenge of integrative genomics. That will be fun, exciting and challenging!

Here’s a short video explaining what the LIBD is about and why Baltimore is a growing city.

ggplot Tutorial

Fri, 21 Jun 2013 00:00:00 +0000

ggplot Tutorial

I liked the following ggplot2 tutorial which is featured in Gabriela de Queiroz’s blog called unbiasedestimator. The tutorial looks very neatly presented and I’m sure that it will be very helpful to anyone just getting started with ggplot2 before they jump into ggplot2: Elegant Graphics for Data Analysis by Hadley Wickham or R Graphics Cookbook by Winston Chang.

The tutorial is very nicely formatted with code in bold highlighting parts that change something in the plot. Overall, the tutorial explains how to use qplot() although it does have a longer example using ggplot() to make survival curves.

Check it out!

unbiasedestimator:

Good tutorial about the R package.

“ggplot2 is an R package for producing statistical, or data, graphics, but it is unlike most other graphics packages because it has a deep underlying grammar. [..]”

- H.Wickham, ggplot2, Use R, DOI 10.1007/978-0-387-98141_1, © Springer Science+Business Media, LLC 2009 -

Keeping in touch (ENAR2014?) and philosophical questions regarding México's future in genomics

Wed, 12 Jun 2013 00:00:00 +0000

The following text is an email I sent to several of my friends from the LCG undergraduate program I studied. There I talk about keeping in touch, I invite them to ENAR 2014, and also talk about some philosophical questions regarding our future.

I’m posting it here because I don’t mind sharing these thoughts and because I don’t have the current email addresses of many former LCG students.

Enjoy

Hola a todos,

Les escribo por el gusto de mantener el contacto, por razones académicas, y también por un rollo filosófico / a futuro.

Contacto

Como ya les dije, me da mucho gusto escribirles. Cuando lo hago sé que generalmente resulta en mails largos, así que intentando ser corto, siento que hoy termina el ciclo de personas que conocí en la LCG. Digo, hoy se gradua la 7ma generación y realmente no conozco las nuevas. Fueron años muy buenos los que viví con todos allá en Cuerna y ojalá podamos mantener cierto vínculo a través de los años. Sé que cuesta. Por ejemplo, no tengo los correos de algunos (o no los encontré vía google), otros los intenté adivinar vía http://www.lcg.unam.mx/titulados y la regla de 8 caracteres, y solo pocos lo siguen teniendo en su página de Facebook (por rollos de seguridad). También soy bastante malo para escribir regularmente (ya sea vía email o mis blogs: académico & más personal).

Este año en especial he disfrutado (y seguiré disfrutando) mucho de ver a algunos aunque sea un par de horas =)

También me gusta mucho cuando me piden ayuda sobre algo, aunque sea lento en contestar :P

ENAR 2014

De lo académico, les quiero avisar que en 2014 van a organizar acá en Baltimore el congreso ENAR (Eastern North American Regional) en marzo del 2014. Es un congreso de bioestadística con una buena parte enfocada a la genómica. Y bueno, muchas cosas hechas en R. Mi jefe está organizando un par de las sesiones así que si tienen sugerencias de quienes invitar avísenme. Ustedes mismos ^_^ (hay un buen número de pláticas por alumnos de posgrado & postdocs), sus jefes, etc. En fin, tal vez les interese el congreso.

De 2014 no hay mucha info aún http://www.enar.org/meetings.cfm, pero allí pueden ver lo que pasó en otros años.

Rollos filosóficos

El rollo filosófico / a futuro tiene que ver con:

¿Qué condiciones piden para trabajar en México?
¿Cómo crear un vínculo más fuerte entre egresados LCG?
¿A qué dedicarse? ¿Academia? ¿Industria? ¿Otra cosa?

Del punto (1) lo he platicado con algunos pero creo que es un tema interesante y muy variado. Hay aspectos que tal vez están fuera de nuestro alcance como la seguridad, y hay otros como el ambiente de trabajo y de la ciencia. El rollo de plazas y los esfuerzos de algunos por crear espacios: CCG tiene algunas ahorita, Palacios & Dávila están haciendo un plan grande para repatriar ex-LCGs, el INMEGEN & el CINVESTAV han estado creciendo, o la idea de hacer un plan nosotros. Ojo, a penas me enteré que para tener plaza de investigador en la UNAM te piden tener un postdoc de 2 años: el chiste es que tengas publicaciones & hayas demostrado que la puedes hacer en la ciencia.

¿Sacrificas potencial académico trabajando en México? Dicho de otra forma, ¿cómo te afecta: el rollo del idioma, el de competir en condiciones que parecen ser más difíciles, o el de tener menos colaboradores cerca para crecer con su crítica & datos de última tecnología? O irse fuera de México (no hay por que ser mártir) y sentirse a gusto con esa decisión. Siempre hay formas de colaborar.

Del punto (2) yo opino que puede ser a través de comentar artículos con gente ex-LCG involucrada (algo así describí aquí), o tal vez vía Más Ciencia por México (no los logré convencer en esto de comentar artículos). Digo, también ayudaría que http://www.lcg.unam.mx/titulados tenga ligas a las páginas actuales de cada quien.

Del (3), en mi opinión cubre también todo el rollo de si queremos vivir con “soft-money” a través del ciclo de donativos (el pay-line es 10% en EUA ahorita por lo de sequester), de la libertad académica (¿es una realidad o una forma de cautivarnos?), y también depende del tipo de industria. Otra forma de ver este punto es ¿cual es el nivel de estrés con el que estamos dispuestos a vivir? ¿A qué aspiramos?

En fin, dan ganas de organizar alguna reunión de ex-LCGs!! Mientras, a seguir con estudios & disfrutar la chamba (investigación o lo que estén haciendo).

Si me preguntan que busco con este correo, es simple:

Mantener el contacto
Invitarlos a ENAR 2014
Promover discusión en estos temas filosóficos / a futuro

Saludos,

Leo

PD Perdón, terminó siendo un mail largo :P

userR2013 data analysis contest: data exploration

Wed, 12 Jun 2013 00:00:00 +0000

Description

The useR2013 conference is organizing a data analysis contest, check the rules here.

They have a package called useR2013DAC with two data sets: one from La Liga and the other one from the Formula 1. Once you download and install the package (available here), you can quickly explore the data using the following R commands:

Data exploration

## Load the package
library(useR2013DAC)

## Explore laliga data
data(laliga)
head(laliga)

##    Season Week               HomeTeam                    AwayTeam
## 1 2008/09    1   Athletic Club Bilbao     Union Deportiva Almeria
## 2 2008/09    1        Atlético Madrid                   Málaga CF
## 3 2008/09    1          Betis Sevilla Real Club Recreativo Huelva
## 4 2008/09    1             CA Osasuna               Villarreal CF
## 5 2008/09    1            CD Numancia                FC Barcelona
## 6 2008/09    1 Deportivo de La Coruña              Real Madrid CF
##   HomeGoals AwayGoals
## 1         1         3
## 2         4         0
## 3         0         1
## 4         1         1
## 5         1         0
## 6         2         1

summary(laliga)

##     Season               Week        HomeTeam           AwayTeam        
##  Length:1900        Min.   : 1.0   Length:1900        Length:1900       
##  Class :character   1st Qu.:10.0   Class :character   Class :character  
##  Mode  :character   Median :19.5   Mode  :character   Mode  :character  
##                     Mean   :19.5                                        
##                     3rd Qu.:29.0                                        
##                     Max.   :38.0                                        
##                                                                         
##    HomeGoals      AwayGoals   
##  Min.   :0.00   Min.   :0.00  
##  1st Qu.:1.00   1st Qu.:0.00  
##  Median :1.00   Median :1.00  
##  Mean   :1.65   Mean   :1.14  
##  3rd Qu.:2.00   3rd Qu.:2.00  
##  Max.   :8.00   Max.   :8.00  
##  NA's   :50     NA's   :50

lapply(laliga, class)

## $Season
## [1] "character"
## 
## $Week
## [1] "integer"
## 
## $HomeTeam
## [1] "character"
## 
## $AwayTeam
## [1] "character"
## 
## $HomeGoals
## [1] "integer"
## 
## $AwayGoals
## [1] "integer"

## Explore formula1 data
data(formula1)
head(formula1)

##   Pos No             Driver             Team Laps        Time Grid Pts
## 1   1  8    Fernando Alonso          Ferrari   49 1:39:20.396    3  25
## 2   2  7       Felipe Massa          Ferrari   49  +16.0 secs    2  18
## 3   3  2     Lewis Hamilton McLaren-Mercedes   49  +23.1 secs    4  15
## 4   4  5   Sebastian Vettel      RBR-Renault   49  +38.7 secs    1  12
## 5   5  4       Nico Rosberg      Mercedes GP   49  +40.2 secs    5  10
## 6   6  3 Michael Schumacher      Mercedes GP   49  +44.1 secs    7   8
##                                         Race Season
## 1 2010 FORMULA 1 GULF AIR BAHRAIN GRAND PRIX   2010
## 2 2010 FORMULA 1 GULF AIR BAHRAIN GRAND PRIX   2010
## 3 2010 FORMULA 1 GULF AIR BAHRAIN GRAND PRIX   2010
## 4 2010 FORMULA 1 GULF AIR BAHRAIN GRAND PRIX   2010
## 5 2010 FORMULA 1 GULF AIR BAHRAIN GRAND PRIX   2010
## 6 2010 FORMULA 1 GULF AIR BAHRAIN GRAND PRIX   2010

summary(formula1)

##       Pos            No                     Driver    
##  Ret    :254   1      :  58   Felipe Massa     :  58  
##  1      : 58   10     :  58   Fernando Alonso  :  58  
##  10     : 58   11     :  58   Heikki Kovalainen:  58  
##  11     : 58   12     :  58   Jenson Button    :  58  
##  12     : 58   14     :  58   Kamui Kobayashi  :  58  
##  13     : 58   15     :  58   Lewis Hamilton   :  58  
##  (Other):848   (Other):1044   (Other)          :1044  
##                    Team          Laps             Time          Grid     
##  Ferrari             :116   55     :125   +1 Lap    :268   1      :  58  
##  Force India-Mercedes:116   56     :121   +2 Laps   :102   10     :  58  
##  HRT-Cosworth        :116   53     : 92   Accident  : 93   11     :  58  
##  McLaren-Mercedes    :116   57     : 80   +3 Laps   : 41   12     :  58  
##  STR-Ferrari         :116   70     : 75   Hydraulics: 26   13     :  58  
##  Lotus-Renault       : 78   52     : 69   Gearbox   : 24   14     :  58  
##  (Other)             :734   (Other):830   (Other)   :838   (Other):1044  
##       Pts          Race               Season    
##         :812   Length:1392        Min.   :2010  
##  1      : 58   Class :character   1st Qu.:2010  
##  10     : 58   Mode  :character   Median :2011  
##  12     : 58                      Mean   :2011  
##  15     : 58                      3rd Qu.:2012  
##  18     : 58                      Max.   :2012  
##  (Other):290

lapply(formula1, class)

## $Pos
## [1] "factor"
## 
## $No
## [1] "factor"
## 
## $Driver
## [1] "factor"
## 
## $Team
## [1] "factor"
## 
## $Laps
## [1] "factor"
## 
## $Time
## [1] "factor"
## 
## $Grid
## [1] "factor"
## 
## $Pts
## [1] "factor"
## 
## $Race
## [1] "character"
## 
## $Season
## [1] "numeric"

I don’t see a specific question that they want you to answer with this data, but if you find one related to data analysis or visualization then join the competition!

Note that you must be attending the conference in order to be eligible to compete.

Reproducibility

```r sessionInfo() ``` ```r ## R version 3.0.0 (2013-04-03) ## Platform: x86_64-apple-darwin10.8.0 (64-bit) ## ## locale: ## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 ## ## attached base packages: ## [1] stats graphics grDevices utils datasets methods base ## ## other attached packages: ## [1] useR2013DAC_0.1-1 knitr_1.2 ## ## loaded via a namespace (and not attached): ## [1] digest_0.6.3 evaluate_0.4.3 formatR_0.7 stringr_0.6.2 ## [5] tools_3.0.0 ```

What does it take to be a good leader?

Wed, 12 Jun 2013 00:00:00 +0000

Leader: scientific or project

In my mind before trying to answer this question I have to define leader. Right now I have two —possibly conflicting— leaders in mind. One is a scientific leader in the sense of a leader in a specific scientific discipline. The other is a leader who can organize and lead projects, either scientific ones (across labs for example) or what I want to call revolutionary projects. With such a grandiose name I am trying to cover the type of projects that can help change a country. However, in reality this second type of leader is closer to administrative roles in leading academic institutions, like the chair of a department.

Good leader

So, how do we say that a leader is a good leader?

For the project leader (second type), I believe that the key is that a good leader gets the job done. The job might have some flaws, but this type of leader can lead a team to produce results. The next aspect you might want to request from a good leader is that they get the job done efficiently (time, resources). However, I would argue that another important characteristic is reducing (minimizing if we were to consider it a mathematical function) internal tensions (frictions).

As for the scientific leader, I struggle more to define it. Is a good scientific leader someone that produces good ideas and publishes them in a timely manner? Is it measured by their ability to secure research funds? Or is it that during their lifetime they had one brilliant idea that changed a field? Or is it just a measure of the number of citations?

Common aspects

I think that one of the most important skills both types of leaders should have is the ability to write clearly. If you cannot write clearly, then you cannot communicate your ideas properly. Without communicating your ideas, you cannot convince anyone that your research work is interesting or that your project is going to succeed.

Additionally, I think that the ability to listen to others and synthesize their ideas is important for both types of leaders. If you cannot listen to others, you will be left alone pretty soon. But you cannot be involved in meetings all day long, so being able to quickly understand and keep the key points others are telling you (what I called synthesize) will be very helpful.

I guess that another important trait is being able to manage some important details secretly but still being able to discuss them with others at some level where you get useful feedback. Said in a different way, you want to test if your idea sounds good but you don’t want to spill the beans and let everyone jump on it before you are ready.

Diverging aspects

Furthermore, I strongly believe that frequent communication helps reduce the internal frictions. You have to tell what is the next step in the plan, keep everyone involved, but you also have to listen to what others are telling you. While I think that frequent communication helps a lot in a lab environment or when leading a project, again I am not sure that it is something a good scientific leader has. But I know that it is something I aspire to accomplish.

I then wonder if good oral communication skills are required for good leaders. I think that the answer is a definite yes for good scientific leaders, who after all frequently expose their work at conferences. But a project leader might not need such type of skill. Sure, they give speeches here and there but probably talk less in front of audiences.

I am certainly separating the ability to talk one-on-one or one-to-a-small-group (like in a table) from good oral communication skills. For me the latter are related to talking in front of audiences while the former are strongly based on the person’s social skills.

How to become a good leader?

That is really the question that I am currently asking myself.

For writing, I think that practicing is very important. That is one of the key reasons why I like to blog and do so in English. As for listening to others, well, I am not sure how to proceed. The secrecy aspect of things is certainly one of my weakest points.

Frequent communication is something that I think I am decent at doing and the key for me is good email managing skills. I am obviously practicing by trying to write shorter (more concise) emails and keeping my inbox as empty as possible following most of the tips given on Inbox Zero for Life by Keith Rarick.

But overall, I wonder if I should take some courses beyond my scientific discipline of interest if I want to be a good leader. Writing? Maybe. I tried a Coursera course and failed to keep up during the comprehensive exam study season. Some kind of managing projects class? Hm…

Maybe you also need to learn to be patient to be a good leader? After all, my motivation takes a big hit when I have to deal with people I would rather not. I am also not so patient with those that complain without proposing alternatives.

Well, I guess that I am at the stage where I am seeking all the feedback I can get.

Reading an R file from GitHub

Thu, 09 May 2013 00:00:00 +0000

Lets say that I want to read in this R file from GitHub into R.

The first thing you have to do is locate the raw file. You can do so by clicking on the Raw button in GitHub. In this case it’s https://raw.github.com/lcolladotor/ballgownR-devel/master/ballgownR/R/infoGene.R

One would think that using source() would work, but it doesn’t as shown below:

```r source("https://raw.github.com/lcolladotor/ballgownR-devel/master/ballgownR/R/infoGene.R") ``` ```r ## Warning: unsupported URL scheme ``` ```r ## Error: cannot open the connection ```

However, thanks again to Hadley Wickham you can do so by using the devtools (Wickham & Chang, 2013 ) package.

Here is how it works:

```r library(devtools) library(roxygen2) ## Needed because this file has roxygen2 comments. Otherwise you get a ## 'could not find function 'digest'' error source_url("https://raw.github.com/lcolladotor/ballgownR-devel/master/ballgownR/R/infoGene.R") ``` ```r ## SHA-1 hash of file is 6c32a620799eded5d6ff0997a184843d7964724a ``` ```r ## Note that you can specify the SHA-1 hash to be very specific about ## which version of the file you want to read in. ```

We can then check that infoGene has actually been sourced:

```r "infoGene" %in% ls() ``` ```r ## [1] TRUE ```

That’s it! Enjoy!

Citations made with knitcitations (Boettiger, 2013 ).

Hadley Wickham, Winston Chang, (2013) devtools: Tools to make developing R code easier. http://CRAN.R-project.org/package=devtools
Carl Boettiger, (2013) knitcitations: Citations for knitr markdown files. https://github.com/cboettig/knitcitations

Reproducibility

```r sessionInfo() ``` ```r ## R version 3.0.0 (2013-04-03) ## Platform: x86_64-apple-darwin10.8.0 (64-bit) ## ## locale: ## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 ## ## attached base packages: ## [1] stats graphics grDevices utils datasets methods base ## ## other attached packages: ## [1] roxygen2_2.2.2 digest_0.6.3 devtools_1.2 ## [4] knitcitations_0.4-6 bibtex_0.3-5 knitr_1.2 ## ## loaded via a namespace (and not attached): ## [1] brew_1.0-6 evaluate_0.4.3 formatR_0.7 httr_0.2 ## [5] memoise_0.1 parallel_3.0.0 RCurl_1.95-4.1 stringr_0.6.2 ## [9] tools_3.0.0 whisker_0.3-2 XML_3.95-0.2 xtable_1.7-1 ```

Using plyr and doMC for quick and easy apply-family functions

Fri, 26 Apr 2013 00:00:00 +0000

A few weeks back I dedicated a short amount of time to actually read what plyr (Wickham, 2011) is about and I was surprised. The whole idea behind plyr is very simple: expand the apply() family to do things easy. plyr has many functions whose name ends with ply which is short of apply. Then, the functions are identified by two letters before ply which are abbreviations for the input (first letter) and output (second one). For instance, ddply takes an input a data.frame and returns a data.frame while ldply takes as input a list and returns a data.frame.

The syntax is pretty straight forward. For example, here are the arguments for ddply:

library(plyr)
args(ddply)
## function (.data, .variables, .fun = NULL, ..., .progress = "none", 
##     .inform = FALSE, .drop = TRUE, .parallel = FALSE, .paropts = NULL) 
## NULL

What we basically have to specify are

.data which in general is the name of the input data.frame,
.variables which is a vector (note the use of the . function) of variable names. In this case, ddply is very useful for applying some function to subsets of the data as specified by these variables,
.fun which is the actual function we want to run,
and ... which are parameter options for the function we are running.

From the ddply help page we have the following examples:

dfx <- data.frame(
  group = c(rep('A', 8), rep('B', 15), rep('C', 6)),
  sex = sample(c("M", "F"), size = 29, replace = TRUE),
  age = runif(n = 29, min = 18, max = 54)
)

# Note the use of the '.' function to allow
# group and sex to be used without quoting
ddply(dfx, .(group, sex), summarize,
 mean = round(mean(age), 2),
 sd = round(sd(age), 2))
##   group sex  mean    sd
## 1     A   F 40.48 12.72
## 2     A   M 34.48 15.28
## 3     B   F 36.05  9.98
## 4     B   M 38.35  7.97
## 5     C   F 20.04  1.86
## 6     C   M 43.81 10.72

# An example using a formula for .variables
ddply(baseball[1:100, ], ~year, nrow)

##   year V1
## 1 1871  7
## 2 1872 13
## 3 1873 13
## 4 1874 15
## 5 1875 17
## 6 1876 15
## 7 1877 17
## 8 1878  3

# Applying two functions; nrow and ncol
ddply(baseball, .(lg), c("nrow", "ncol"))

##   lg  nrow ncol
## 1       65   22
## 2 AA   171   22
## 3 AL 10007   22
## 4 FL    37   22
## 5 NL 11378   22
## 6 PL    32   22
## 7 UA     9   22

But this is not the end of the story! Something I really liked about plyr is that it can be parallelized via the foreach (Analytics, 2012) package. I don’t know much about foreach, but all I learnt is that you have to use other packages such as doMC (Analytics, 2013) to actually run the code. It’s like foreach specifies the infraestructure to communicate in parallel (and split jobs) and packages like doMC tailor it for specific environments like for running in multi-core.

Running things in parallel can then be very easy. Basically, you load the packages, specify the number of cores, and run your ply function. Here is a short example:

## Load packages
library(plyr)
library(doMC)

## Loading required package: foreach
## Loading required package: iterators
## Loading required package: parallel

## Specify the number of cores
registerDoMC(4)

## Check how many cores we are using
getDoParWorkers()
## [1] 4

## Run your ply function
ddply(dfx, .(group, sex), summarize, mean = round(mean(age), 2), sd = round(sd(age), 
    2), .parallel = TRUE)

##   group sex  mean    sd
## 1     A   F 40.48 12.72
## 2     A   M 34.48 15.28
## 3     B   F 36.05  9.98
## 4     B   M 38.35  7.97
## 5     C   F 20.04  1.86
## 6     C   M 43.81 10.72

In case that you are interested, here is a short shell script for knitting an Rmd file in the cluster and specifying the appropriate number of cores to then use plyr and doMC.

#!/bin/bash 
# To run it in the current working directory
#$ -cwd 
# To get an email after the job is done
#$ -m e 
# To speficy that we want 4 cores
#$ -pe local 4
# The name of the job
#$ -N myPlyJob

echo "**** Job starts ****"
date

# Knit your file: assuming it's called FileToKnit.Rmd
Rscript -e "library(knitr); knit2html('FileToKnit.Rmd')"

echo "**** Job ends ****"
date

Lets say that the bash script is named script.sh. Then you can submit it to the cluster queue using


qsub script.sh

This is what I used to re-format a large data.frame in a few minutes in the cluster for the #jhsph753 class homework project.

So, thank you again Hadley Wickham for making awesome R packages!

Citations made with knitcitations (Boettiger, 2013).

Revolution Analytics, (2013) doMC: Foreach parallel adaptor for the multicore package. http://CRAN.R-project.org/package=doMC
Revolution Analytics, (2012) foreach: Foreach looping construct for R. http://CRAN.R-project.org/package=foreach
Carl Boettiger, knitcitations: Citations for knitr markdown files. https://github.com/cboettig/knitcitations
Hadley Wickham, (2011) The Split-Apply-Combine Strategy for Data Analysis. Journal of Statistical Software 40 (1) http://www.jstatsoft.org/v40/i01/

Epi vs Biostat Kickball match Spring 2013

Wed, 24 Apr 2013 00:00:00 +0000

This past Saturday the Epi and Biostat troops met for another fun kickball match. Obviously Biostat beat Epi, yup I know: again! This time the score was 15-8 (according to our bookkeeper and captain John) or 12-8 (according to some in Epi).

There was a hint of a surprise at the beginning when Epi scored two runs in the top of the first inning. However, the tide changed back with a homerun by Rumen. Sadly, one of the Epi players got injured and carried out of the court in that play. Rumen also pulled his quad with the big hit and was limited for the rest of the match.

From that inning on forth we saw both teams having fun kicking the ball as far as we could or aim for in between the defensive lines. There were plenty of sacrifice hits, some occasional errors, but overall we had a lot of fun!

Both teams came prepared to show their colors: them in red us in purple with some face paint for the sport battle (thanks to Aaron). However, the Epi crew did surprise us by bringing a big grill to the park and lots of food!

At the end of the match, we all mingled together and enjoyed the nice (a bit chilly) day outside in the company of some drinks and food.

Some of us then continued our journey at Kislings where we played other games that involve loads of cups and some ping pong balls ;)

You can view all the pictures here. If you have any other pictures that you want to share, send them my way!

"Do analytics really tell the whole story?"

Sun, 31 Mar 2013 00:00:00 +0000

“Do analytics really tell the whole story?" by Vic Ketchman explores how analytics is used nowadays in the NFL draft. The entry point is the "Moneyball" movie and Ketchman’s piece is mainly a digested interview to Tony Villiotti from draftmetrics.com

According to him:

What is analytics? It’s the accumulation of meaningful patterns in data, for the purpose of using that data to predict future results.

I’m not a fan of the wording used, but well, the point they make is that they use data to predict the future.

My main issue with this article is that after the previous quote Ketchman pretty much describes some of the data. Description of the data—in my opinion—is part of what we call EDA: Exploratory Data Analysis. The data is interesting, but there are really not many predictions made.

I’m also concerned by how some of the data is presented. For example, is the 37.1 percent rate of starts by first-round picks really different form 35.5 for the teams with losing records? Plus, it’s data from only a single year! So I think that it’s not enough to actually answer any question.

To end my comment, Ketchman asks:

How do you like those analytics?

I don’t like them much. Sure, some of numbers presented are interesting but the ‘analytics’ are far from being great. Though I bet Villiotti has more interesting results that are only seen by the NFL teams.

Have you been 'relative stupid'?

Sun, 31 Mar 2013 00:00:00 +0000

I enjoyed reading “The importance of stupidity in scientific research" by Martin A. Schwartz which I learned existed through @hmason and @simplystats.

I found the point of how it’s normal to feel stupid in academia and specially in Ph.D. programs to be illuminating. But Schwartz clarifies that there are other kinds of stupid:

we don’t do a good enough job of teaching our students how to be productively stupid – that is, if we don’t feel stupid it means we’re not really trying. I’m not talking about 'relative stupidity’, in which the other students in the class actually read the material, think about it and ace the exam, whereas you don’t. I’m also not talking about bright people who might be working in areas that don’t match their talents. Science involves confronting our `absolute stupidity’.

I don’t know about you, but I have certainly been ‘relative stupid’ at times.

And yes, we have to confront our ‘absolute stupidity’. But to me, graduate school is also about learning how to be super efficient with your time. That implies being highly organized, learning how to canalize your distractions, and finding sources of constant motivation. For example, I now read more stats/R/research blogs as part of my set of distractions and have considerably decreased how many sport news I read.

I also struggle with the internal challenge of doing great at school, but then also ‘having a life’. So yes, at times I have been ‘relative stupid’ but also had a great time. After all, I no longer need to ‘ace’ all my exams.

Great commentary on sequestration's impact on research! National media should talk about this and YOU should read it!!!

Thu, 28 Mar 2013 00:00:00 +0000

Today Jeffrey T. Leek and Steven L. Salzberg published a paper commentary in Genome Biology today titled “Sequestration: inadvertently killing biomedical research to score political points” (Leek & Salzberg, 2013) which I think is a must read for anyone. Seriously!

I do not mean anyone involved in research, or all scientists. I mean, this commentary should be in the national media. Why?

Well, let me approach the technical side first. You might think that anything that appears in a scientific journal—despite any efforts to make it accessible to the general public—will rely on words whose meaning is mostly only understood by scientists. That is not the case in this commentary: it is a dual letter meant to be read by those in Congress, but it is also an educational commentary for the general public.

The main reason why you should be reading this commentary is that the consequences of the ‘sequester’ are going to affect you. So if you are interested in your future and the well-being of those who you care for, then you should read it. And if you don’t know what the sequester is and how it will impact research, well, that’s another reason why you should read this commentary. Plus you might want to look at this (serious) comic from PhD comics (© Cham, 2013) to get an overall idea. Note that it was published before sequestration hit in.

Going back to the commentary piece by Leek and Salzberg, I can imagine someone refuting like this:

Hey, but I don’t live in the United States so it doesn’t affect me.

That is true in a sense because you will likely be affected by your own country’s policies more directly, and specially in policy topics that have short term impact. Nevertheless, any breakthrough made by U.S.-based research for the most part (aka, when politics doesn’t get in the way) will reach you. After all, Leek and Salzberg cite (Alivisatos et al in The Atlantic, 2013) where the following statement is made:

Nobel Prize-winning economist Robert Solow has calculated that over the past half century, more than half of the growth in our nation’s GDP has been rooted in scientific discoveries – the kinds of fundamental, mission-driven research that we do at the labs.

The claim that it affects other countries is just a generalization of the previous result and what I would consider some common sense. If this is not enough to attract your interest, then you should take a look at Salzberg’s previous comment “A breakthrough cure for acute leukemia?” that is a showcase example of successful biomedical research funded by the same institutions being hit by sequestration.

I hope to have convinced you to read Leek and Salzberg’s commentary by now. So let me talk a little bit about the things that I liked the most.

Most of all, I like the tone they used because this is not a silly matter and while it may sound as alarming as the boy who cried wolf, the reality is that the wolf does exist and will visit you. So while no visible effects have been seen from the sequester this month, that doesn’t mean that you can just ignore this problem. It is like when you throw a stone in calm water: just a few small ripples are seen at the beginning, but they reach far away. In other words, it will take some time to actually feel the negative effects.

Overall, I consider Leek and Salzberg’s work a wake up call to politicians and you. Either you the researcher, but most importantly, you the citizen who cares about the future.

Some, specially those who are major supporters of military programs, might disagree with the whole comparison of the F-35 plane which has an estimated cost of $400 billion to the National Institutes of Health (NIH) annual budget of around $31 billion (Leek & Salzberg, 2013). But to me this is just incredible!

To end my comments, I have to say that I am surprised that Leek and Salzberg’s commentary is behind a paywall. I thought that it would be an open-access piece. After all research articles in Genome Biology are open-access, but this is a commentary so it is not considered a research article. To their credit, Genome Biology does offer 30-day free trial subscriptions. But I am afraid that Leek and Salzberg will lose many readers due to this reason. Hopefully, you will feel motivated enough to go through the whole trial subscription process, or maybe Genome Biology will make an exception for this commentary.

**Update: Genome Biology changed Leek & Salzberg’s commentary so as of March 28th it is now open-access (I wrote the post late on the 27th).

Finally, are you not incredulous to see this situation happen? Shouldn’t the debate be about spending more money in research now that what was spent in the past? The whole sequestration topic is alarming, but the fact that the budget for research hasn’t increased in years is shocking. Oh wait, you are giving Mexico a chance to catch up to the mighty U.S. in research!!! The whole talk in Mexico about catching up with Brazil or India should be about the U.S. now! (Sadly, Mexico has a lot of catching up to do…)

Citations made with knitcitations (Boettiger, 2013) and the post was written in the Rmd format powered by knitr (Xie, 2013).

(2013) The Sequester Is Going to Devastate U.S. Science Research for Decades. The Atlantic http://www.theatlantic.com/politics/archive/2013/03/the-sequester-is-going-to-devastate-us-science-research-for-decades/273925/
Carl Boettiger, knitcitations: Citations for knitr markdown files. https://github.com/cboettig/knitcitations
Yihui Xie, (2013) knitr: A general-purpose package for dynamic report generation in R. http://CRAN.R-project.org/package=knitr
Jorge Cham, U.S. Budget Sequestration Explained. http://www.phdcomics.com/comics.php?f=1561
Jeffrey T Leek, Steven L Salzberg, (2013) Sequestration: Inadvertently Killing Biomedical Research to Score Political Points. Genome Biology 14 10.1186/gb-2013-14-3-109

New Citi ThankYou ballon TV commercial looks like a ripoff of Hello Kitty in Space

Wed, 27 Mar 2013 00:00:00 +0000

I don’t know about you, but I think that this new “Citi ThankYou Cards” TV commercial is trying to ride the popularity train from the “HELLO KITTY IN SPACE” video.

Hm… it looks like a ripoff, smells like a ripoff, tastes like a ripoff… is it a ripoff?

Maybe it’s just flattery, maybe it’s imitation, or maybe it’s copyright infringement. What do you think?

Predicting who will win a NFL match at half time

Sat, 23 Mar 2013 00:00:00 +0000

It was great to have a little break, Spring break, although the weather didn’t feel like spring at all! During the early part of the break I worked on my final project for Jeff Leek’s data analysis class, which we call 140.753 here. Continuing my previous posts on the topic, this time I’ll share the results of my final project.

At the beginning of the course, we had to submit a project plan (more like a proposal) and in mine I announced my interest to look into some sports data. At the time I included a few links to Brian Burke’s Advanced NFL Stats site (Burke). At the time I didn’t know that Burke’s site described in detail a lot of the information I would end up using.

My final project had to do with splitting NFL games by half and then use only the play-by-play data from the first half to predict if team A or B would win the game. My overall goal was to have some fun with sports data which I had never looked at, but then also try to come up with something I would personally use in the future. So, why split games by half? I personally would like to know if I should keep watching a game or not at half time. Having a tool to help me decide would be great, and well, if the team I’m rooting for has high chances of losing or winning, ideally I would switch to doing something else. A related question that I didn’t try to answer is which half is worth watching? This would be a meaningful question if you only have time to watch one of them.

To truly satisfy my goals, it wasn’t enough to just build a predictive model. That is why I also built a web application using the shiny package (RStudio and Inc., 2013). It was the first time I did a shiny app, but thanks to the good manual and some examples on GitHub from John Muschelli like his Shiny_model it wasn’t so bad. I thus invite you to test and browse my shiny app at http://glimmer.rstudio.com/lcolladotor/NFLhalf/. It could be improved by adding some functions that scrape live data for the 2013 season so you don’t have to input all the variables needed by using the sliders. Anyhow, I’m happy with the result.

The entire project’s code, EDA steps, shiny app, and report are available via GitHub in my repository (lcollado753). While the details are in the report, I’ll give a brief summary here.

Basically, I summarized the play-by-play data for all NFL games from 2002 to 2012 seasons as provided by Burke (Burke, 2010). I used some of the variables Burke uses (Burke, 2009) and some others like the score difference, who starts the second half, and the game day winning percentages of both teams. After exploring the data, I discarded the years 2002 to 2005. Then, I trained a model using the 2006 to 2011 data and did some quick model selection. Note that I’m not doing the adjustment by opponent the way Burke did it (Burke, 2009-2) in part because I was running out of time, but also because the model already uses the current game winning percentages of both teams to consider the two team’s strength. I evaluated the model using the 2012 data and after seeing that it worked decently enough, I trained a second model using the data from 2006 to 2012 so it can be used for the 2013 season. These two trained models are the ones available in the shiny app I made.

In the report, I didn’t include ROCs—a big miss—so here they go. The code I will show below is heavily based on a post on GLMs (denishaine, 2013). The code below is written in a way that you can easily reproduce it if you have cloned my repository for the 140.753 class (lcollado753).

First, some setup steps.

## Specify the directory where you cloned the lcollado753 repo
maindir <- "whereYouClonedTheRepo"
## Load packages needed
suppressMessages(library(ROCR))
library(ggplot2)

## Load fits.
## Remember that 1st one used data from 2006 to 2011
## and the 2nd one used data from 2006 to 2012.
load(paste0(maindir, "/lcollado753/final/nfl_half/EDA/model/fits.Rdata"))

Next, I make the ROCs for both trained models using the data that they were trained on. They should be quite good since it uses the same data to build the model that it will then try to predict.

## Make the ROC plots

## Simple list where I'll store all the results so I can compare the ROC plots later on
all <- list()

## Construct prediction function
for(i in 1:2) {
	## Predict on the original data
	pred <- predict(fits[[i]])
	
	## Subset original data (remove NA's)
	data <- fits[[i]]$data
	data <- data[complete.cases(data),]
	
	## Construct prediction function
	pred.fn <- prediction(pred, data$win)
	
	## Get performance info
	perform <- performance(pred.fn, "tpr", "fpr")
	
	## Get ready to plot
	toPlot <- data.frame(tpr = unlist(slot(perform, "y.values")), fpr = unlist(slot(perform, "x.values")))
	all <- c(all, list(toPlot))

	## Make the plot
	res <- ggplot(toPlot) + geom_line(aes(x=fpr, y=tpr)) + geom_abline(intercept=0, slope=1, colour="orange") + ylab("Sensitivity") + xlab("1 - Specificity") + ggtitle(paste("Years 2006 to", c("2011", "2012")[i]))
	print(res)
	
	## Print the AUC value
	print(unlist(performance(pred.fn, "auc")@y.values))
}

```r ## [1] 0.8506 ```

## [1] 0.8513

Both ROC plots look pretty similar (well, the data sets are very similar!) and have relatively high AUC values.

Next, I make the ROC plot using the model trained with the data from 2006 to 2011 to predict the outcomes for the 2012 games.

## Load 2012 data
load(paste0(maindir, "/lcollado753/final/nfl_half/data/pred/info2012.Rdata"))

## Predict using model fit with data from 2006 to 2011
pred <- predict(fits[[1]], info2012)

## Construction prediction function
pred.fn <- prediction(pred, info2012$win)

## Get performance info
perform <- performance(pred.fn, "tpr", "fpr")

## Get ready to plot
toPlot <- data.frame(tpr = unlist(slot(perform, "y.values")), fpr = unlist(slot(perform, "x.values")))
all <- c(all, list(toPlot))

## Make the plot
ggplot(toPlot) + geom_line(aes(x=fpr, y=tpr)) + geom_abline(intercept=0, slope=1, colour="orange") + ylab("Sensitivity") + xlab("1 - Specificity") + ggtitle("Model trained 2006-2011 predicting 2012")

## Print the AUC value
print(unlist(performance(pred.fn, "auc")@y.values))
## [1] 0.816

The steps in the curve are more visible since it is using less data. It also seems to be a little less good than the other two, as expected. This is clear when comparing the AUC values.

Finally, I plot all curves in the same picture to visually compare them.

names(all) <- c("train2011", "train2012", "pred2012")
for(i in 1:3) {
	all[[i]] <- cbind(all[[i]], rep(names(all)[i], nrow(all[[i]])))
	colnames(all[[i]])[3] <- "set"
}
all <- do.call(rbind, all)

ggplot(all) + geom_line(aes(x=fpr, y=tpr, colour=set)) + geom_abline(intercept=0, slope=1, colour="orange") + ylab("Sensitivity") + xlab("1 - Specificity") + ggtitle("Comparing ROCs")

Both ROCs with the trained data (train2011, train2012) are nearly identical and both are slightly superior to the one predicting the 2012 games.

Overall I am happy with the results and while some things can certainly be improved, I look forward to the NFL 2013 season. Also, remember that Burke publishes his winning estimated probabilities from week 4 onward (The Fifth Down Blog). So you might be interested on comparing the probability at half time versus his estimated probability which is calculated before the game starts. I mean, maybe you could use the difference between the two to have an idea of how unexpected the first half was. After all, if a game falls outside the pattern it might be worth watching.

Citations made with knitcitations (Boettiger, 2013).

lcolladotor, lcollado753. GitHub https://github.com/lcolladotor/lcollado753/tree/master/final/nfl_half
denishaine, (2013) Veterinary Epidemiologic Research: GLM – Evaluating Logistic Regression Models (part 3). denis haine http://denishaine.wordpress.com/2013/03/19/veterinary-epidemiologic-research-glm-evaluating-logistic-regression-models-part-3/
Advanced NFL Stats. http://www.advancednflstats.com/
(2010) Advanced NFL Stats: Play-by-Play Data. http://www.advancednflstats.com/2010/04/play-by-play-data.html
(2009) Advanced NFL Stats: How the Model Works–A Detailed Example Part 1. http://www.advancednflstats.com/2009/01/how-model-works-detailed-example.html
(2009) Advanced NFL Stats: How the Model Works–A Detailed Example Part 2. http://www.advancednflstats.com/2009/01/how-model-works-detailed-example-part-2.html
By BURKE, Brian Burke - The Fifth Down Blog - NYTimes.com. The Fifth Down Â» Brian Burke http://fifthdown.blogs.nytimes.com/author/brian-burke/
Carl Boettiger, knitcitations: Citations for knitr markdown files. https://github.com/cboettig/knitcitations
RStudio , Inc. , (2013) shiny: Web Application Framework for R. http://CRAN.R-project.org/package=shiny

"I am a writer" exercise

Thu, 21 Mar 2013 00:00:00 +0000

I do not have a clear memory of when I started to write or in which language it was. My first written words might have been in English since I lived in Boston (USA) three years during my early childhood. By age five I was back in Mexico and that is where I am sure I wrote my first full homeworks. During elementary school, I changed languages once more—this time to French. By middle school, I started to be interested in two new types of languages. One was mathematics which I liked, but which I didn’t consider till much later. The other was related to computers as I learnt the very basics of HTML—that’s all I know so far. In college—having reverted back to Spanish and English—and in my current stage in graduate school, I am a writer because I write—I mainly typeset using LaTeX—my homeworks, code in R, and summarize findings in reports. For the past year, I have been using Fellgernon Bit to practice writing and hopefully improve my skills. Furthermore, for me the process of writing helps me clarify my thoughts and organize them before attempting to communicate them. Sometimes it works, others it doesn’t. Finally, I am a writer because it is crucial in the academic environment to be able to communicate through the printed word. This is tricky because sometimes you want to be very short, direct but not leave anything important out, like when emailing a professor. Other times, you have to be very precise and clear yet tell an interesting story such as when writing a scientific report.

Overall, I consider myself a writer in training and would like to improve. But as with everything, practice is key. That’s a big part of why I blog and why I’m enrolled in this course.

And so begins English Composition I

Thu, 21 Mar 2013 00:00:00 +0000

This week started the English Composition I: Achieving Expertise course (Comer, 2013) that I have been looking forward to.

I am not sure yet how long I will last, but I hope to enjoy it as much as I can. Plus, it should help me with my posting and other writing areas. While I last in the course, I plan to publish my writings in the blog too. So you will hopefully see me be more active here.

As it is important to cite when writing, I have also figured out how to do so automatically in Rmd files. For that I learnt how to use knitcitations from the GitHub instructions (knitcitations) and a explanatory post (Boettiger, 2013).

knitcitations is great, but it kind of struggles with some pages. That is why I modified my template in FBit by writing my own citing function for pages where citep fails. Here is the code:

## I made my own citing function since citep() doesn't work like I want to with
## urls that are not really pages themselve like part of a GitHub repo.
mycitep <- function(x, short, year=substr(date(), 21, 24), tooltip=TRUE) {
	tmp <- citep(x)
	res <- gsub("></a>", paste0(">", short, "</a>"), tmp)
	if(tooltip) {
		res <- gsub("\\?\\?\\?\\?", year, res)
	}
	res
}

## You already saw an inline working example in the post itself.

Carl Boettiger, (2013) knitcitations. Lab Notebook http://www.carlboettiger.info/2012/05/30/knitcitations.html
cboettig, knitcitations. GitHub https://github.com/cboettig/knitcitations
Coursera. Coursera https://www.coursera.org/course/composition

Commenting scientific papers

Mon, 11 Mar 2013 00:00:00 +0000

I’ve been thinking about commenting papers in blog posts. I did a few some long time ago, but now I’m thinking of doing this activity more systematically. There are several reasons why I’m thinking of doing this, say for 1 paper a week. It has the obvious advantage of forcing me to read a paper in depth per week. At the same time, I want to learn more from others. See what I like in other papers and maybe avoid some mistakes. There are two main lines of papers that I would be posting about. Anything that is somewhat close to my research (genomics, RNA-seq, biostatistics, bioconductor, visualization) and anything done by my undergrad peers from LCG-UNAM. I don’t think that there is a compilation of papers from LCG students despite many of us doing research all over the globe —Mexico, US, Canada, Denmark, France, England, Germany, Switzerland, Austria, Australia to name a few countries. Maybe compiling a list of papers with contributions from LCG students is a task for Más Ciencia por México which seeks to promote science in Mexico. But I would be happy to learn what others are doing and in a way keep in touch academically.

Another reason in favor is that blogging helps me practice my English. And writing helps me organize my ideas.

But, the question remains, if you systematically comment papers, what would you comment on?

I think that I should state my opinion of the paper in different areas. Kind of like doing a review. First, try to summarize the paper. Next, was the scientific objective clear? They did answer the main question? Then, given the nature of my Ph.D. program, I think that I should try to comment on any statistics used in the papers. This certainly includes the plots and reproducibility. If they included tools (software), I could take a quick look at it. Then, I can end with stating the main things I liked. Maybe I could come up with some scoring mechanism to rate the paper.

You can think of other aspects to talk about of a paper. For example, in what way did it help it’s field? But, I don’t think that I can answer this for many papers outside my research area. It would all be speculation. I guess that I could use Google Scholar to see who cited the paper and maybe comment on it’s impact that way. For the LCG papers, I could point out how the LCG students contributed.

Or maybe I could take the more educational route. But that’s very time consuming as I can see from La Ciencia explicada's highly detailed posts.

Anyhow, if I have something clear in mind is how I would implement it. I’m thinking of making a GitHub repository and writing my comments using Rmd and knitr. Then posting them here using Markdown. It should be easy to then have a template post and fill in the gaps after reading the paper.

The risk of using a template is that the comments will start to look boring. That’s why I might add a more free section, or change things up a bit.

If you have any ideas, let me know!

Analyzing SimplyStatistics visits info

Sat, 09 Mar 2013 00:00:00 +0000

Recently we had to analyze the data of the number of visits per day to SimplyStatistics.org. There were two goals:

Estimate the fraction of visitors retained after a spike in the number of visitors
Identify (if any) any factors that influence the fraction estimated in 1.

For me it was a fun project in part because I like SimplyStatistics but also because I think that finding the answers to the questions would be interesting and help understand the readers of that blog.

Sadly, I didn’t work on it much. We had lots of stuff due that week, but well, I’m happy enough with the analysis I did. My own report is hosted here and this is the pdf file of the report itself.

Half joking with other students, I said that I basically did t-tests. Hopefully I can work on changing this tendency with the pile of recommended books I’ve been acquiring but not really reading through. Except for the ggplot2: Elegant Graphics for Data Analysis and the R Graphics Cookbook. Sounds like spring break will be fun :P

Kind of related to this, Jeff Leek announced yesterday that he is going to compile a list of student blogs that have something to do with statistics and data. He added a link to my blog which is why I saw a large peak of Fellgernon Bit’s visitor data. After all, when doing the data analysis described above I played around with the data from Fellgernon Bit and now know that at a minimum posting drives visitor’s into sites (which sounds obvious, but maybe you get random traffic) —see fig 1 of the report.

Had Jeff done so before, I could have a point estimate (but without being able to say something about the uncertainty of it) that SimplyStatistics has 142 visitors that read the posts AND click on the links. Maybe using the info from Hilary’s and Alyssa’s blogs we could have an estimate with some measure of uncertainty, but only for March 8th.

Alfred: a must for any Mac user

Fri, 15 Feb 2013 00:00:00 +0000

At the beginning of the semester, I decided to go hunting for Mac apps that would help me be more organizing and/or enjoy my Mac even more. After all, I was using the basics –with multiple spaces– and had only customized my favorite editors.

It turns out that Alfred is an excellent app. The free version can get you a lot of mileage and save you lots of time by typing alt + space, then entering the keyword you want to search in Google. Or alt + space, image, then the query for Google Images. Or alt + space, find, the parts of the name of a file you want. Or alt + space, in, something you want to find inside a file. I love the alt + space, define, something I want to find in the dictionary. I know that dictionaries are just around the corner [a bookmark away!] but still, thanks to Alfred I now look up words WAY more frequently than what I did before. I mean, just not having to move my hands away from my keyboard and doing a ton of stuff is just great =)

There are plenty of other default searches that come with Alfred’s free version. Another thing that I love is using it to do system commands like lock the screen, or send my computer to sleep.

Try it out!

Second cultural mixer today!

Fri, 15 Feb 2013 00:00:00 +0000

Liking "Inbox Zero for Life"

Thu, 14 Feb 2013 00:00:00 +0000

I’ve been using the “Inbox Zero for Life" strategy for a few weeks, and I think that it’s been payed off for me in this short span.

As it’s stated in that long guide, one of the major concerns you might have is that it could end up as just changing a current problem for another one. I think that so far, that hasn’t been the case for me. Sure, my starred emails is not 0, but it stays at a steady number and doesn’t increase as my inbox (even with priority inbox) did.

I also have a few filters in place that pick up the emails that I will most likely never read. For example, advertising emails and school wide announcements.

One point I’m not sure that I buy is the whole psychological effect of having an empty inbox. But whether or not that’s true, I certainly didn’t know that the gmail app in the iPhone/iPad has a weird smiley that looks like a sun telling you something like: “Your inbox is empty. Have a nice day!”

Credit goes to Hilary for finding telling us about this strategy.

Sharing my work for "Advanced Methods III"

Wed, 13 Feb 2013 00:00:00 +0000

This semester I’m taking the live version of the Data Analysis class by Jeff Leek. His more popular version of the course is available through Coursera.

One of the things that Jeff promotes is reproducibility and sharing code. I share that tendency and thus created a Git repository for my homework and code for the class: lcollado753. I’m hosting it with GitHub to try it out since I started with Mercurial via Bitbucket.

Part of me would love it if everyone in the class had their own Git repositories. I mean, this class involves lots of practice exercises and there are plenty of R packages and functions that others use that I would like to learn. As I don’t see this happening, I think that it would be great to list the packages/functions you think could be interesting to others at the end of the write-ups. However, this involves sharing the reports and I don’t know if that will happen.

But maybe I didn’t get the instructions Jeff gave correctly the first time. Listening into his week 2 talks from the Coursera course, I get that he wants our reports to be reproducible. The idea is great, but sometimes I get lots in the technicalities of finding the best fit for our situation. Aka, something we can all do that is worth the time for small scale projects that we have a couple of days to complete and most likely will be finishing the day before they are due. For now we might stick to sharing zip files with the report + summarized data set (it has be small enough to be sharable by email).

I’m pretty happy with hosting my stuff at GitHub. One blunder I made in the first data analysis report is that I completely forgot to say in it that I have the code in GitHub :P Oh well, next time!

I feel that I also have lots to improve regarding how to tell a story in a report. Plus, for this first project I mainly did some exploratory data analysis without much stat analysis.

Overall, I’m quite excited with this course =) and I think that I’ll learn a ton on methods to analyze data AND how to actually implement them. Plus, I’m currently trying to learn ggplot2 as you can see in that first report. Also, I made it with knitr instead of Sweave =)

Adding youtube videos in pdfs, html reports and html presentations

Wed, 05 Dec 2012 00:00:00 +0000

I got a question today on how to add a video to a beamer pdf presentation. Well, I had never done it, but I got curious enough to google around for a bit and here is the end product.

One way of doing it is using the media9 tex package. For this to work you need to have the latest version of texlive (or miktex). Then, it’s quite straight forward to include the video. The issue is that you have to open the pdf with Acrobat Reader 10+ (9 something works. I think that it’s 9.4.1+ but well, the point is that you need an updated version). You will also need a live web connection to actually show the video. An alternative (if you have the video file) is to convert it to swf and embed it.

Here is my tex example file with the corresponding pdf output compiled using pdflatex. Remember that you need Acrobat Reader to actually see the video in the pdf.

After doing this, I just wanted to make some quick examples on how you can add a video to an html report using markdown (well, you add the video itself using html). I did it via knitr and Rstudio.

Example Rmd file with corresponding md and html * outputs from “knit HTML” in Rstudio.

Finally, the more interesting thing you can do is an html presentation which requires pandoc. Again, I used knitr to get things started and, while at it, add some R and math.

Example Rmd file with corresponding md and html * output from “knit HTML” in Rstudio. The slides are created from the md file using pandoc into this html ** file (main output in this case).

Hopefully they’ll be useful to anyone venturing into embedding youtube videos in their presentations.

* These two files are very similar.

** html presentation.

Introduction to R and Biostatistics (2012 version): presentation

Mon, 12 Nov 2012 00:00:00 +0000

To follow my Introducing R and Biostatistics to first year LCG students (2012 version) post, you can now find the presentation online from my site either in presentation format, in a single webpage format, or the raw Rmd file. To prove the point that publishing to RPubs is super easy, you can also find the single webpage format over there. I also like how you can comment and share in RPubs.

One of the challenges of giving a presentation to first year students is finding the balance between introducing them to cool things you are doing in your work and actually giving a talk that they can follow. I thought about this and ended dropping anything related to my work.

My presentation was split pretty much in two parts. First, I wanted to promote some philosophical discussion about what is statistics. Second, I gave a brief overview of what you can do with R. Or more exactly, what they should be able to learn to do even if they become wet biologists.

While planning this presentation, I knew that I wanted to give the new students a flavor of the three different currents in statistics. I aimed to improve my 2011 explanations now that I’m taking the Foundations of Statistical Inference course. I’m happy with the result and I think this is greatly due to Royall’s diagnostic test example.

Another key point that I wanted to emphasize was that RStudio is the way to go if you are new to R. It is very straightforward to use, plus it is nicely interegrated with knitr.

I decided to use R Markdown (Rmd) for the first time, after seeing how easy Markdown really is compared to using LaTeX and Beamer. However, when it got to doing the presentation I have to say that I was a bit dissapointed by how some things just break when using the R Markdown to Markdown to HTML presentation pipeline —using pandoc for the last step. For example, the math breaks when using mathml or mathjax at times (like after adding an iframe for a youtube video), so I had to use webtex which doesn’t look as nice.

If you are interested in the commands, I used the “Knit HTML” button in R Studio [equivalent to running from R: library(knitr); knit(“filename.Rmd”)] and then ran the following command:

pandoc -s -S —webtex -i -t dzslides intro_R_Biostat_LCG_2012.md -o intro_R_Biostat_LCG_2012_slides.html & open intro_R_Biostat_LCG_2012_slides.html

I was originally aiming to have a single Rmd file to produce an HTML presentation and a Beamer presentation. However, controlling the pictures in the Beamer output proved to be challenging. While I had a work around, the final problem was the math part. By the time I realized this it was too late —I just dropped the Beamer presentation. Googling, even the author of knitr acknowledges that the best input for PDF output is still LaTeX.

In the end, I’m happy that I got the HTML presentation done using R Markdown and briefly introduced it to the first year students. The basics of knitr are very easy to learn and I’m hoping that it got some of them curious enough to try it.

Next thing in line: prepare 5 questions for the students.

me: Bad rm, don't delete stuff I didn't want to delete! (rm: well, I do what you tell me to do!)

Wed, 07 Nov 2012 00:00:00 +0000

When Sandy was in town at some point I started doing some of my research work, but I shouldn’t have. I basically did a silly mistake and erased files that take a long time to compute.

Prior to being here, I had an alias in my bash profile like this:

alias rm=’rm -i’

But when I setup my bash profile here I googled a bit to find what was the best common solution to avoid deleting stuff you shouldn’t be deleting. That’s when I found the following stackoverflow entry: `alias rm=“rm -i”` considered harmful?

One of the answers suggests using rmi instead of rm for the alias. The idea is that you will never expect that rm will run interactively in other machines, which could potentially be disastrous.

Being “smart”, I set up my alias to be rmi. However, I started using rm (without expecting it to be interactive) and just ignored the alias.

Problem is that I ran a more command with a pattern, saw that the files were empty, then deleted them with rm. Next, I wanted to check a different pattern, used the up arrow key and instead of editing the more command, I edited the rm command. When I noticed it, it was too late.

Luckily, I only deleted some output files that I can recover. But it was a pretty basic mistake. And yes, the files were not backed up.

Anyhow, I will now stick to my rmi alias and hopefully avoid running into this kind of pitfall ever again.

Got acknowledged in a paper ^^

Wed, 31 Oct 2012 00:00:00 +0000

During the weekend while I was talking with a friend and former colleague, I realized that my name was mentioned in the acknowledgments section of a paper :) I haven’t been much in touch with what’s been happening back home, so this was a nice surprise.

The paper is: Genetic changes during a laboratory adaptive evolution process that allowed fast growth in glucose to an Escherichia coli strain lacking the major glucose transport system by Aguilar et al. It was published in BMC Genomics in 2012.

Introducing R and Biostatistics to first year LCG students (2012 version)

Tue, 30 Oct 2012 00:00:00 +0000

On Friday November 9th I’ll be giving a talk to the first year students from the Undergraduate Program on Genomic Sciences (LCG in Spanish) during their “Seminar 1: Introduction to Bioinformatics” course. It’s just like I did a year ago as I documented in my post Introducing Biostatistics to first year LCG students.

Well, this time I’ll change things a bit. I’m allowed to require the students to read 2-3 papers before my talk to introduce them to my field. I’ll do so, but in a more peculiar way by requiring them to listen in to a few videos I selected. So, without further ado here are the three required “papers”:

Here is “paper 1" (~30 minutes). The goal is to introduce you to the basic workings of R and also to great sources of R videos.

First, learn to install R (watch it in full screen).

Or you can also watch any of the two following videos:

Next learn about RStudio and why it’s a great place to start (watch it on hd and fullscreen).

Now you are ready to learn how to create a variable in R. Use RStudio instead of the R GUI to do so.

Next, learn the super basics about the basic R plot system.

Now you are ready to learn about how to use the combine function.

Next, learn about data.frame type of objects

and how to add new variables to them.

Next up is learning how to find help.

Almost there. Now check how to change your current working directory.

Finally, learn how to install and load a package in R.

If you are more curious regarding the origins of R check the next video (not part of “paper 1”).

Next, “paper 2" (~39 minutes). The goal here is to get a feeling of how you can use R to create plots.

First start with this demonstration of the basic R plotting tools (called “base graphics”). It does in enough level of detail of how the basic plotting system works and how you can customize the colors, layout, etc. For the purpose of getting used to the tool, I recommend that you follow this video using RStudio. Also, you’ll want to watch it in 720p.

Now check the demo for plotting with the lattice package. This is more advanced, but it should also be more illustrative of the power you have with R. Plus it shows how we can expand the functionality of R by using packages contributed to the community and freely available for us to use.

Finally, “paper 3" (~28 minutes). This is the first lecture from a course by Brian Caffo in which he goes over the definition and overall motivation behind Biostatistics. It should be much more fun to watch than reading a review paper in the area.

Now, for those motivated to learn more, I recommend some of my own posts summarizing information that can be useful to you.

Sources:

http://www.twotorials.com/ by Anthony Damico
Youtube videos by Roger Peng
Youtube videos by Brian Caffo

Super basic Illustrator: event invitation

Wed, 24 Oct 2012 00:00:00 +0000

Amanda and I are organizing a cultural student mixer for students in our department. One of the things we needed to do was invite everyone to attend. I like using computers, but graphic design is not something that I’m too excited about, but I still wanted to try something out.

My goal was to show an image of a bowling lane where the Earth (as the ball) knocks down “statistical pins”. We recently covered least squares in our class, so I felt that the analogy of “minimizing our cultural differences” fit perfectly.

I’ve used Photoshop in the past, but after reading (or parts of) Photoshop, Illustrator, or Indesign? by Chris Takakura I decided to use Illustrator. The first thing I tried to do was to crop the Earth from the following image which I found using Google Images with the “large” option (I didn’t want small pics that would look bad if enlarged).

Source

But it turns out that you don’t “crop” in Illustrator. You use “clipping masks” as explained here. It took me some time to figure this out, but it was easy once I used the “ellipse tool”. I was expecting a visual shape, but I got a circular “path”.

Next, I found a good bowling lane image:

Source

With this, it was easy to “place” the image (I found the term from this tutorial). After resizing everything and moving the Earth image to the front (arranging it), I followed this tutorial to place text on a path. This was quite easy.

Finally, I just wanted to add a green checkmark for the drinks + snacks + beers message. That was easy using the following image:

Source

After some revisions, the end result was the following ad which we sent via email to the other students:

This is as far as my graphic design (well, putting together) skills go :P

Visualizing colors()

Fri, 19 Oct 2012 00:00:00 +0000

The other day I learnt about the existance of the colors() vector in R which specifies all the character-based colors like “light blue”, “black”, etc. So I made a simple plot to visualize them all. Here’s the code:

mat <- matrix(1:length(colors()), ncol = 9, byrow= TRUE)
df <- data.frame(col = colors(), 
	x = as.integer(cut(1:length(colors()), 9)),
	y = rep(1:73, 9), stringsAsFactors=FALSE)
plot(y ~ jitter(x), data = df, col = df$col,
 	pch=16, main = "Visualizing colors() split in 9 groups",
 	xlab = "Group", 
	ylab = "Element of the group (min = 1, max = 73)",
	sub = "x = 3, y = 1 means that it's the 2 * 73 + 1 = 147th color")

And the plot:

Bitbucket revamped

Tue, 09 Oct 2012 00:00:00 +0000

Bitbucket announced their new “look” today. The goal is to make it more team friendly but I guess that they also wanted to make it look fresh.

For example, the overview page now has a quick summary:

That can be useful coupled with the simpler navigation tabs. But I think that the best of the new tools is the ability to comment at a given commit at any line change.

This gives a new dimension when working with a team. It’s independent of the version control system you are using, so this can be a drawback in a sense as you need to log into Bitbucket to see the comments. I guess that if you keep the main explanations inside commit messages, this new comment tool can be helpful when reviewing the code and/or solving merges.

Bitbucket thought about how to make these comments more visible, so they show as updates in the overview page. That’s great, otherwise you would have to go to each commit and check if there is anything new out there. Plus, there is an RSS feed for those of us that prefer to use these instead of browsing to a webpage to check if there is an update.

To foment the discussion when solving merges, they also have some new tools inside the “pull requests”:

Overall, these changes make me happy and want to stay with Bitbucket. Though most of my collaborators use Github and the idea of having to get a new account is a HUGE wall. You wouldn’t think it is after having accounts for lots of other stuff, right?

Well, remember that you can log into Bitbucket using OpenID:

<3 Bitbucket

Why aren't all of our graphs interactive?

Mon, 08 Oct 2012 00:00:00 +0000

During the last pre-happy hour seminar, Karl Broman talked about Why aren’t all of our graphs interactive? I didn’t know, but a few years ago Karl worked in the department and clearly promoted beer-drinking and is the heart of the department. I’m a fan of our pre-happy hour seminars since you have a get to listen to good/fun talks over a beer or two.

But I’m also a fan of reproducible research and useful graphics. I do most of this by using Sweave (for reproducibility) in LaTeX documents and with the R packages lattice, car, and plotrix, and some ggplot2 (I should use it more).

Karl made his presentation using html (definitely check it out!) and inserted pretty interactive graphics. His talk got me really interested and I definitely need to pick up a few tools. For example, asciidoc or R Markdown can be useful for making html documents with R code. Specially if you want to write a report and you don’t want to deal with Sweave/Latex when making plots (can be a pain to know where they’ll show up).

For the interactive side, D3 (and other tools Karl listed) can be useful to learn. But I might put this on a hold for some time. Maybe I’ll wait and see what others in the deparment are developing for R-D3 and embedding interactive plots in pdf files.

I don’t think that it will be long before interactive plots make it to the journals. Specially for their web versions. Though, I still think that if you are showing a 3D plot, as the author you will have to give a few default views where you can clearly see something that you want to talk about instead of having the reader find that sweet spot.

One problem that I don’t think has been solved yet is reproducible research on a cluster. Karl and others mentioned make as well as having if/else clauses where you either show the output or a cleaned up version of the code that you used to generate the output.

Overall, there are many tools and tips I can learn from Karl. And I’m sure that I’m not the only one! Hopefully he’ll give tips on where to start (nothing is more tedious than reading UNIX man-files).

An Online Bioinformatics Curriculum

Sun, 07 Oct 2012 00:00:00 +0000

Last week I talked about online courses in my JHSPH-Biostat through Coursera post. Now I’m back to comment on An Online Bioinformatics Curriculum by David B. Searls. Sur Herrera pointed out this paper to me, and I have to say that if you are considering learning bioinformatics online it will be very useful to you. David Searls first goes through a history recap of online (free) courses. Notably, in the last year Coursera and other startups offered their first courses. MIT has also evolved and now offers MITx courses, where MIT does give certificates. For example, I’m a bit interested in the Introduction to Computer Science and Programming 6.00x course. It aims to cover a wide variety of topics which can be nice for a review/learning and is Python-based.

The main body of David Searls’ paper is a huge list of summaries for the main courses out there for bioinformaticians. He covers several tracks depending on what subarea of bioinformatics you are interested in. For each summary, he recommends an specific course to take along with the main reasons why he prefers it over other options. If you have ever looked into OCW, Coursera, etc; you know this is a great resource. After all, there are lots of options for some courses like calculus and it can be a time drain to look through them before deciding which to take. This summary is the best part of the paper.

David Searls ends it with a conclusion section and some tips on how to make the best of the available resources. How far can free online education go? Well, obviously for informatics it can go almost all the way compared to wet lab biology. Plus, you have to be organized/dedicated/motivated and a good self-learner. Even with all that, you still need good ideas to use as learning projects. If you are looking for one, I would take a look here and here where Jeff Leek lists some of his project suggestions.

02/27/12 PHD comic: 'Inspired by true events'

Wed, 03 Oct 2012 00:00:00 +0000

02/27/12 PHD comic: ‘Inspired by true events’

Totally what happened to me on Monday night! (whenever I saved this post as a draft :P, but it’s happening again now)

Check all your accounts for emails at once in Gmail

Tue, 02 Oct 2012 00:00:00 +0000

Do you use Gmail as your primary email account? Primary, what? Well, I’m sure that you have your personal account somewhere and your work or university account too. You can either log into each email interface or you can integrate them to a single one.

Gmail makes this easy as you can go to the “Settings -> Accounts and Import” to set up Gmail to pull all of your email into it. In that same page you can configure Gmail to send email through your work/university accounts or at least camouflage it.

That is all great, but as you will soon learn, Gmail pulls the emails from your other accounts at random intervals. It tries to do so in an intelligent way by pulling more frequently during the times of the day that you get more emails from those accounts. But you’ll soon realize that some emails get pulled an hour later.

Maybe this is what you want. Specially if you get TONS of emails and you don’t want anyone to expect immediate replies from you, thus promoting more intelligent emails from them and hopefully reducing your email traffic load.

But maybe you don’t like this behavior. That’s where the ”Multiple Inbox” Google Lab comes in.

You can find it by going to “Settings -> Labs”. “Multiple Inbox” is designed to help those that have multiple Gmail accounts. But it also changes the behavior of the “refresh” button.

Once you enable “Multiple Inbox”, the refresh button will automatically look for new emails in ALL of your accounts. Thus you will no longer have to go to “Settings -> Accounts and Import” to manually look for new emails by clicking on the “Check mail now” links.

Note that “Multiple Inbox” creates automatic panes if you are using the default inbox. If you don’t like them, you can simply go to “Settings -> Multiple Inbox” and clear the default searches.

Enjoy!

[Edited: minor word change]

JHSPH-Biostat through Coursera

Mon, 01 Oct 2012 00:00:00 +0000

Have you heard of online education? If you are in the US or Mexico I’m sure that you have seen some ads about online universities. Well, that’s not the type of education I’m talking about. I’m talking about free high-quality education.

For some years, the top option has been the Open Courseware (OCW) organized under the Open Courseware Consortium (OCWC). Back in 2009 I was pushed my undergrad (LCG-UNAM) to design and teach OCW-compliant courses. I even taught a course on R/Bioconductor and thought of it as a pilot OCW course. The first seven classes were video recorded. But that project hit a wall because many of the biology professors used slides that heavily relied on copyrighted material. For OCW courses you have to own the copyright of the material that you use (or get permission), so just the idea of having to re-do all the diagrams and figures was overwhelming. This hasn’t stopped some big universities like MIT from publishing OCW-compliant courses.

Recently there’s been talk of the new horse in the race: Coursera. What is it? Well, according to themselves:

We are a social entrepreneurship company that partners with the top universities in the world to offer courses online for anyone to take, for free. We envision a future where the top universities are educating not only thousands of students, but millions. Our technology enables the best professors to teach tens or hundreds of thousands of students.

Through this, we hope to give everyone access to the world-class education that has so far been available only to a select few. We want to empower people with education that will improve their lives, the lives of their families, and the communities they live in.

So far, the motivation is similar to the OCW movement. However, one very big difference is that Coursera does offer certificates. Something which OCW courses do not. For example, MIT-OCW says:

OCW is not an MIT education.

OCW does not grant degrees or certificates.

OCW does not provide access to MIT faculty.

Materials may not reflect entire content of the course.

Coursera courses do provide the entire content of the course. Well, this is slightly tricky since some professors use the same base material in the university-in-class courses but expand it beyond what is available through Coursera. Thus in a sense Coursera are more accesible courses with lesser requirements than the in-class versions. But compared to OCW, you have homeworks (which are graded) and can communicate with the faculty through the use of forums.

One advantage of OCW courses is that you can look at them whenever you want. For Coursera ones you have to sign up (and thus register to their system) and they are open for certain periods of time.

Currently, the Biostatistics Department at JHSPH is offering three courses through Coursera. These are Computing for Data Analysis by Roger D. Peng, Mathematical Biostatistics Boot Camp by Brian Caffo, and Data Analysis by Jeffrey Leek. The first two are introductory courses to using R and Biostatistics, respectively. I’m taking the in-class versions and highly recommend them to anyone that wants to get started in either topic. They both involve youtube videos and practice exercises. The videos themselves are great since they rehearse what they are going to say, used a high-quality audio recording room, tuned the audio, and included highlights in the slides so you can follow them easily. Right now you can go and sign up for these two courses!

The third one, Data Analysis, is more advanced and I’ll take it in-class next year. In addition, for now the sign up is closed for 2012 (you can go ahead and save a spot for 2013).

All of these courses have a couple thousands students registered, which is great! I’m sure that the great majority will greatly benefit from them. To finish my post, I’ll leave you with their short introduction videos, which will tell you more than what I can via text!

Enjoy!

Learning about social networks through an interactive presentation

Fri, 28 Sep 2012 00:00:00 +0000

During this week’s journal club meeting Hilary Parker (homepage, blog) led the session on “Identifying influential and susceptible members of social networks”. Were there some speakers or why did she “lead the session”? By this I mean that Hilary tried a very different (and interesting) format this time. Instead of giving a talk, not a formal one like at seminars, she prepared a short presentation (publicly available here) that begins showing a 20 minute video. This video is by the author of the paper where he presents the key points of his research at another conference. The goal of this format was to get us to speed and hopefully provoke enough discussion to make the meeting highly interactive. Plus the author does a great job in his presentation.

Now, given that it’s a biostat journal club, Hilary included some slides to explain the general Cox Proportional Hazard model before showing some of details used in the paper in question.

I think that the change of format was a step in the right direction. Hopefully others will follow.

About the paper itself, the topic is interesting since it shows a different view of the “data science” vs biostat discussion. The presenter is trying to convince computer scientists that they need to do some statistics too. Over here, we are poking our heads at whether we need to learn some computer science.

In addition, the author is in a different setting than academia or industry, which are commonly the two options. He is at NYU, and academic institution, but he is working closely with the industry (thus getting access to interesting data) and might be getting some consultation money along the way. Anyhow, given the recent talk from Amy Heineike on Quid (more in my previous post) there is a growing interest among students to learn more about the industry environment.

I made a couple of comments during the discussion, which might be completely wrong. One is that I feel that in academia we care much more about bias and removing sources of error and it seems that in industry that’s not the main point. There you care more about making something useful which might be biased. You try to minimize it, but the judge are the clients.

The second one is that in industry options like getting a larger sample are much more feasible. In the video, at some point the author shows that people eventually joined the app after getting massively spammed. Increasing the exposure is easy in this case, but imagine a public health survey that is carried out door by door. Increasing the number of houses visited is way more expensive.

The point is that the club meeting followed an interesting format, social networks seem fun to analyze, and industry vs academia is kind of a hot topic in our department right now.

—

Thanks to Hilary for publicly sharing her journal club presentation.

Quid @Biostat-JHSPH

Fri, 28 Sep 2012 00:00:00 +0000

Just like most scientific departments, we have a seminar (weekly over here) where very bright people come to us to talk about their work. Being a Biostatistics department, we mostly get faculty from other Biostatistics departments from universities to talk to us. This week was quite different. Amy Heineike from Quid gave us a talk describing their product, which fits perfectly in what is now called “data science”. You can see Amy at the end of the table in the picture below.

So what is Quid? It’s a start up tech company that provides either their software or reports derived from it that help big companies (a) analyze a field, (b) look at what the competition is doing, (c) take informed decisions (helpful for marketing). The short video below describes Quid in a more general way, check it out!

As Amy Heineike described in her talk, the three common decision-taking pathways are:

Someone follows their own intuition. Say a big shot that thinks he knows where the world is going.
Someone with decision power asks others to generate reports for her/him. That is, lots of manual work where some read, consult others, etc then they summarize the information in a report.
Similar to the above one where lots of people gather the information, then a program is run and the decision is pretty much made by the computer.

The Quid paradigm is to use the computer to gather all the information and then have a human(s) look at a network with a very cool 3D tool to assimilate the information and decide themselves. The argument is that the human brain is very powerful for visual pattern recognition and can out-perform computers.

At first I felt that you can do the network part with a software like Cytoscape which I find to be very powerful for network analysis. But the pipeline used by Quid is much more extensive and it’s an all-in-one bundle.

Another key argument in favor of Quid is that most of the information shared is done in a list format. Like google search results, powerpoint bullet points, your facebook feed, etc. But who came up with the ranking? How are things related? That’s when you need a network representation.

I recommend taking a look at their technical overview page where they have the main steps outlined. But needless to say, they depend strongly on the natural language processing early steps. Their 3D tool looked very interesting and I love to play with it. Amy Heineike actually poked us by showing a video of a short session using the software that was designed so we would want to have a go with Quid. I, as many others, were hooked! Sadly, Quid’s software is not the kind that academics can go buy for now.

I found the example using “synthetic biology” as the query to be pretty interesting. Sadly I don’t have a picture, but one of the features that seems very powerful is when you change to a 2D display. In it, you have the time on the X-axis and the number of articles (well, any kind of input file Quid can use) on the Y-axis. By clicking on a point (which corresponds to a node in the network 3D environment) you can then visualize all the connections that are directly linked to it. Thus you have a scatterplot with a 2D network on top of it. That information can be really useful to understand the flow of information. The specific example was how someone proposed years ago that a specific kind of application was possible, time later grants on the subject were announced, and more close to the present he got a grant, then other grants and results were publicized.

Now, Quid has some flaws. For instance, one hot question was how to control the threshold that determines whether two nodes are connected or not. The answer was something like this: experts in their fields have validated the results for queries related to them. Not very convincing for a biostat crowd. Another one was how to control/remove/correct bias. Amy Heineike replied that you need to learn where the data used by Quid is like. For example, when looking at companies the number of news articles mentioned is linked to how efficient/big their public relations office is.

Nevertheless, Quid’s product is very interesting. Plus, I feel that part of our tool-box as Biostatisticians is visualizing data in ways that allow us to understand what is going on. As for working at Quid or doing anything alone the line, we definitely need to learn more about computer science. After all, you need incredibly fast algorithms and code to work with enormous data sets.

——

PS Amy Heineike might develop a Pubmed scrapper for Quid. Meaning that Quid would be able to access citations data. Then it would be very cool to use a few “seed” papers that you are interested in to find the complete history behind them and any other papers similar to them. There might another group out there working in your field that you don’t know about! Which I think happens more frequently that what you think. Specially if you don’t look abroad.

——

Edit: I had completely forgotten that I had read about Amy Heineike before in her SimpleStatistics interview. There’s more about her in this video and in The Phenomlist.

Introducing Git while making your academic webpage

Mon, 24 Sep 2012 00:00:00 +0000

Last week I gave a presentation during our computing club on how to use git (a version control system). I used as a motivating example the first steps of creating your own academic webpage. The goal was to make it interesting to both new students (who might have been more interested on the webpage part) and older students (for whom version control should be a must). The slides and all the material is publicly available through the following Bitbucket repository: https://bitbucket.org/lcolladotor/html_git_intro/overview. You can access the slides by clicking on “Source”, “slides” and then “html_git.pdf”.

For the talk, I tried to make it more interactive but at the same time I wanted to make sure that the material could work for reference in the future. For example, I added a commands.txt file so anyone following me could easily copy-paste the commands. By the way, for Git Bash in Windows, you paste stuff by using the insert key instead of the usual “ctrl + v” shortcut.

I’m posting about it as it could be useful to other people, so feel free to share it.

Enjoy!

Version control: need to learn Git

Wed, 05 Sep 2012 00:00:00 +0000

I consider myself a fan of using version control for bioinformatics/biostatistics (or any text based, like code) project. Yet my knowledge of version control systems is quite limited. I’ve used Mercurial for some time, but I haven’t ventured much beyond the basic commands and some GUIs for merging.

I don’t recall how it all went, but I remember reading that Subversion (SVN) was much better than CVS. Also, the Bioconductor project uses SVN. Before that I really learnt how to use SVN, someone from the Bioconductor devel list pointed me to Git/Mercurial. Around the same time I read “A quick guide to organizing computational biology projects" by William Noble, which further convinced me to start using a version control system. I "educated" (a tiny bit) myself on the topic with Wikipedia’s entries on revision control and distributed revision control.

I wasn’t sure whether to use Mercurial or Git, but at the time Bitbucket only supported Mercurial repositories. It felt pretty easy to use, specially after reading the guide whose examples covered pretty much all I needed. By the way, I highly recommend using Bitbucket now (whether for Mercurial or Git repositories) as they offer unlimited private repositories to anyone with an academic email account.

Now for my Advanced Methods class by Brian Caffo (check out his Mathematical Biostatistics Boot Camp Coursera free online course) I need to learn how to use Git. That lead me to check some Git vs Mercurial posts such as:

Curiosity and another reason lead me to watch the video from above. It helped me to understand the basic differences between Git and Mercurial, plus it reassured me that skipping SVN was a good thing. I might still need to learn SVN properly, but at least through Git-SVN or HgSubversion it seems that I can dodge the bullet.

I’ll come back once I’ve tried out Git, but for now it seems that SourceTree will be a great tool to have. It works with Bitbucket and Github (free for open source, gotta pay for private repositories).

To finish this post, if you are new to the topic you should check out:

Well, even without knowing much about these tools you probably already use some kind of version history thanks to Dropbox and Google Docs.

PS I found lots of stuff here.

Homepage photos: which cheek you show says something?

Thu, 30 Aug 2012 00:00:00 +0000

John Bohannon wrote Scientists’ Photos Reveal Their Inner Mr. Spock which I found funny in a science-curious kind of way. I hadn’t thought about the message you could be sending depending on which cheek you show. I had only noticed that people like to show more serious-looking photos of themselves and sometimes make it an obviously funny one. I have to say too that some of the pages from my co-workers don’t even show a picture. I guess that’s a more recent trend to make it more “professional” looking. Dunno.

I have to guess that dead center-looking pictures are rare. At least when you look at Rachael Jack’s one it stands out. It reminds me of passport pictures. For my new academic site I choose one that doesn’t look serious, I wanted a fun one! Plus I make it obvious which country I’m from ^^.

For some reason this doesn’t surprise me:

while English language and literature scholars showed the left cheek and engineers showed the right, psychologists were more difficult to categorize.

Is science cool?

Mon, 27 Aug 2012 00:00:00 +0000

Adam Ruben shares his answer to whether science is cool or not. Since “cool” depends on the current trends, this question is set on the recent landing of Curiosity. It’s the landrover that recently landed on Mars and has been sending some pictures back.

Anyhow, check his answer here. I liked this part:

If the Mars landing draws students to science, it won’t be because they witnessed science doing something cool. It’ll be because they witnessed science doing something human, something genuine, something legitimately appealing on its own merits. Something scientific.

Picture taken from his commentary piece.

Setting up your computer for bioinformatics/biostatistics and a compedium of resources

Thu, 23 Aug 2012 00:00:00 +0000

Jumping on the train set by Hilary Parker “The Setup (Part 1)" and Alyssa Frazee “my software/hardware setup”, I’m going to share my setup and hopefully add something new. They both did a great job already, so make sure you read their posts!

I have some experience with all three main OS: Windows, Linux and Mac. That being said, I know some of the basic stuff for each but I surely use Google very frequently to get help. I used to have a dual Windows / Linux (Ubuntu) set up but now I have a Windows laptop/desktop (it’s a monster :P) at home and I’m happy working with my Mac.

I’m going to start by mentioning the software I use(d) in each OS and then add some other tools that I really like.

Windows

Text editor: Notepad++. It outperforms Notepad by light years! A must for me is the “View -> Word wrap” option. I would definitely go to “Settings -> Preferences -> New Document/Default Directory” and change the new document format from Windows (Dos) to Unix. This will save you time later when you want to work on a Unix system like the cluster. If you didn’t, you can change a specific document’s EOL (end of line) by using “Edit -> EOL conversion -> UNIX format”. Another feature that I like is the “Search -> Replace…” which allows you to use regular expressions (like Perl).
Statistical software: R of course! It’s best to do a custom installation and choose a directory without spaces in it. That will help later (further below). If you want to be convinced to join the R community read Data Analysits Captivated by R’s power and R You Ready for R? or just take a look at what others are using. Remember that the R project is open source, free and easy to contribute to. If you end up choosing Excel as your statistical software, well, there is no hope for you!!
R code editor: Notepad++ with NppToR. For a long time I used Emacs modified to work with Windows by Vincent Goulet available here. It works great and saves you quite a bit of setup time. XEmacs is another option that a friend of mine used, but it never convinced me. Anyhow, I ended up changing from Emacs to Notepad++ with NppToR because I could:

Force quit R (and not lose code changes) in case I crashed R by doing something stupid like printing something huge or w/e :P
Access help pages in a separate window. I’m sure you can do it too with Emacs, but I was just lazy to configure it.
Shorter shortcuts
Later on I found the NppToR to PuTTy feature which is very useful.
You can create an R syntax dictionary (or something like that) in Notepad++ which will scan all your R packages and add the function names so they are colored when you type them. Also Notepad++ will auto-complete some function names and show you the arguments. Great stuff! (Forgot the name, so… google it :P)

SSH: PuTTY. As said before, works well with Notepad++ and NppToR.
SCP: WinSCP. There are others that work too like Filezilla but well, WinSCP does the job well.
PDF viewer: Adobe Acrobat Professional. I’m using the X version now. I like how I can highlight, underline, cross out, free hand, sticky note, combine files into a single pdf, combine pdfs, and change the highlight colors easily. It also has a change tracker (kind of like Word has). I’ve seen other use PDF Annotator which is available for free for Hopkins students. Anyhow, I simply love Acrobat for reading papers.
LaTeX: MiKTeX. For writing TeX files I used either Emacs or Notepad++. There is another software which has drop down menus and the like called WinEdt. I got used to typing LaTeX from scratch, well, I have a template.Rnw somewhere. Oh yeah, I always use Sweave when writing TeX files (even if I don’t use R).
R reports: Sweave by Friedrich Leisch, one of the champions of reproducibility! To learn more about Sweave first read this pdf by Nicola Sartori. This is another Sweave demo by Charles J Geyer. Check out this great Windows Sweave troubleshooting page by John D Cook.
Building R packages from source. You will definitely need Rtools installed. I would also install QPDF which can be used by R to compress your pdf files, which is a good thing if you want to have a small-sized tarball. Last but not least, check out Building R packages for Windows by Rob J Hyndman.
Learn to modify your PATH! Check 1.3 from the previous link by Rob J Hyndman. If you are going to use Sweave, it’s best to add to your PATH the path for the directory containing your Sweave.sty file so that you won’t need to copy it to every single directory. This is why it pays off to do an R custom installation and put it in C:/R/R-current-version or something like that instead of C:/Program Files/ bla bla with spaces. It used to be more important a few years ago. Also, I created a sw.bat file and put it somewhere where my PATH would find it. That sw.bat file ran Sweave, pdflatex twice, then bibtex and finally opened the pdf file.
PDF viewer for LaTeX files. I only learnt about pdf sync and the like a year ago. You should google how to set this up with SumatraPDF (Adobe Acrobat doesn’t work!).
Version control: Mercurial. It’s very easy to use and you can get an account at Bitbucket.org with unlimited number of private repositories if you have an academic email. Even if you are not doing a collaborative project, you will love using a version control system! It will clean up your directories very nicely and will help you become more organized. Learning a few commands is nothing compared to having lots of files with _v1 v_2, etc at the end. Check out the Mercurial guide to get started. Note that for windows instead of customizing your .hgrc file you will customize a mercurial.ini file.
Presentations: both PowerPoint and Beamer (normally with Sweave too). Rarely I use Google Docs for this.
Office: Either Microsoft Office or OpenOffice (free).
Poster creator: PosterGenius (academic discount price). It was very easy to use and I would surely give the free trial version a go. It adds a watermark, but well, you will appreciate the time you save compared to using PowerPoint. I guess that Adobe Photoshop is another option, but I’ve only used it to edit photos here and there, not to make a whole poster. The most I did was create this.
To de-compress RAR files: WinRAR.
Anti-spyware: Avast (free version). I normally keep it in silent (gaming) mode so it doesn’t show pop ups.
CCCP codec pack which includes the Media Player Classic. Great for watching video files and dumping the crappy Windows Media Player.

Linux (Ubuntu)

Ubuntu provides Linux distributions that are very user friendly and that look much like Mac OS does now. You’ll find it easy to run multi-core programs which were a pain to do with Windows. Beware that even if you use Ubuntu you will need to learn stuff like how to compile. Also, please check before you install that your computer is supported. For example, some laptops with very new video cards might not work properly. That being said, with Ubuntu you will feel very at ease working in an area like mine (genomics) because a lot of the software runs in Linux (normally in a cluster, but you can test in your lap).

You will want to check and/or keep for reference LINUX Essentials by Thomas Girke (more from him below), Learn Linux in 10 minutes, The Linux tutorial, and Linux vi editor tutorial.

* If you are going to use Linux (Ubuntu) at some point you will want to compile something from source and find out that you are missing a dependency. That’s when I google, then use : 1. apt-cache search something 2. sudo apt-get install something

Note that you will frequently need the yyyy-devel version which includes c headers and stuff that you need to compile.
You will find a lot of things through the package installer (forgot what it’s called). Learn the pseudonym for your Ubuntu version so you select the appropriate version of the software in case that you are downloading it from another place.
SSH/SCP: terminal commands :) I just wanted to mention that rsync is a nice command for synching folders (recursively too) between your computer and say the cluster.
Version control: Mercurial again. The configuration file is .hgrc not mercurial.ini
Text editor: Nedit or Emacs. Vi when doing in-terminal modifications.
You can get R through aptitude or if you want the very latest (or a devel version) you’ll have to compile it. The first time you will have to install plenty of dependencies, but it’s good practice.
Office: go with OpenOffice.
LaTeX: install the texlive distribution. I normally get everything so that later when I’m trying to use a TeX package I won’t have to go install it (which is what MiKTeX does for you in Windows).
Video player: VLC.

Mac

Terminal: iTerm2. Mac comes with a native terminal, but iTerm2 has other nice functions like tabs and more options to customize it. Check this for free color palettes. I like the Homebrew one from here.
LaTeX and R editor: Aquamacs is a version of Emacs that works great. However, as I discussed in the Windows section I’m moving away from Emacs. Well, to be honest, I don’t want to put the time to learn how to customize Emacs properly and do amazing stuff with it like Kasper does. Recently (since June) I’ve been using TextMate. It has this thing called “Bundles” which provides different hotkeys depending on the file you are editing. Meaning that for Rnw files you can Sweave them directly there and for R files you can either send the code to R or to the terminal (much like Notepad++). The one thing is that it is not free BUT there is a 2.0 alpha release available on github that you can compile. This lengthly discussion can be worth reading if you want to know more about the 2.0 version and the future of TextMate. Someone said there that Sublime might be replacing TextMate but I haven’t looked for any R integration in it. Anyhow, I liked how TextMate included an auto-spell checker that recognizes Sweave/LaTeX code from the box :)
Package installer: MacPorts. It’s kind of similar to aptitude from Linux but it’s Mac only. Note that you will definitely need to get XCode.
PDF viewer: also Adobe Acrobat Pro for the reasons mentioned previously.
PDF viewer for LaTeX: TeXShop which I think comes with the MacTeX distribution. It has the forward sync that Alyssa mentions in her blog.
Text editor: TextWrangler. Has several of the functions I talked about in the Notepad++ section like search and replace with regular expressions. It definitely outperforms the native text editor.
SCP: Cyberduck. I haven’t tried others, but it works and I’m happy with it. I also use the terminal to push/retrieve files like I would do in Linux. Same for ssh and Mercurial.
Version control: Mercurial. Note that you might have to add a site key (like bitbucket’s) to your .hgrc file so it doesn’t complain when pushing files.
Productivity: My Little Pomodoro available from the Mac app store. I love it for following the Pomodoro technique (you can use any other timer that you like) which Hilary introduced me to. It works like a charm when you are under stress and need to be productive. After all, I’m prone to escape the stress and distract myself, so this helps me keep my distractions limited. I’ve also found that when I’m stuck in a problem and I take the 5 min break thinking about something else, well, the machine keeps working and when I come back from the break I have a new idea to try out.
Video player: VLC (includes codecs).

Other stuff

Browser: I used to loveMozilla Firefox (it has a nice sync functionality) but I’ve moved to Google Chrome. It’s kind of a shame that Google started to compete with Mozilla, but oh well bye bye 2007. I use Chrome because it works a tad bit better with other Google tools, but that’s it. It also syncs your bookmarks. Both work great and Opera is still my favorite backup browser. I guess anything but Internet Explorer and Safari.
Learning R. I would definitely check Thomas Girke Programming in R page and Frank McCown’s Producing Simple Graphs with R tutorial. I have my own share of R related slides here.
Learning Bioconductor. Thomas Girke again wrote a great resource for learning how to use Bioconductor for analyzing high-throughput sequencing data files. Bioconductor hosts packages for other technologies/problems, so I would also look at it’s own help pages like the Workflows section. I also like Peter’s R programming pages, specially the heatmap section. I have my own share of Bioconductor related slides here.
Learning LaTeX. I learnt the hard way I guess… I learnt by comparing Sweave files and their output and seeing what changed if I modified the code. Nowadays, I would very highly recommend that you first check the How to Use LaTeX short series of exercises/files by Andrew Matchett. The Not so short introduction to LaTeX is a great resource. For very specific symbols, check the Comprehensive LaTeX symbol list.
Using a SGE (Sun Grid Engine) cluster. For some basic commands look here. For the Hopkins cluster, definitely read this. Finally, for running array jobs check this.
LaTeX and math. You should definitely read the wiki books page for this topic. I kept going back to it over my first year at Hopkins when I really needed to learn all this. The theorems page is nice, but not a must. Same for Common TeX/LaTeX errors.
Figures in LaTeX. Here is a basic overview but the wiki books page for the topic is a must check.
Accents in LaTeX. Check this blog post by “Bugs and Solutions”.
Blogging R code: Pretty-R. I haven’t really used it but it surely looks pretty!!
Cloud storage: Dropbox works great and tons of iPad apps have an option to backup to it which works great with my note-taking apps. Google Drive and others are also around.
Paper (biobliography) organizer: Zotero is amazing! I simply love it :) I pull the bibliography from pubmed or the magazine page itself and to avoid any hassle, I have a “papers” folder in my Dropbox where I only organize them by last name. Then if I want to find something, I go to Zotero and look use it’s great search function. I rarely use it to annotate webpages and I hear that it can now upload files to the cloud. Anyhow, I first used it in my Windows/Ubuntu dual setup. You can use it as a Zotero Firefox plugin or as Zotero stand-alone with Zotero Connector (Google Chrome for example). Finally, Zotero can export your bibliography into a BibTex file :)
Blog: Tumblr. Some like WordPress better, but I like how Tumblr is not only a blogging platform but also a social media tool. I’ve written several posts before on how to customize your blog and other blog related tools. But a must in my point of view is to get your RSS feed “burnt” with feedburner. It has lots of interesting tools and is much better than a plain XML RSS feed.
Notes: Notability iPad app. Great stuff and doesn’t blow up on you (aka, doesn’t lose your notes) like NotesPlus did to me. Anyhow, note-taking in my iPad with auto-cloud backups greatly changed my classroom experience. THere are other apps like this one out there.
I use Voice Recorder HD iPad app for recording lectures. It’s useful when you miss something the professor went over quickly and you are trying to understand it later on.
Email: Gmail with keyboard shortcuts enabled. I also use “Canned Responses” from the Google Labs to specify my signature. I have an academic one, one for Mexico, etc.
Send emails later: Boomerang for Gmail.
Calendar: Google Calendar.
Task manager: Google Tasks from within Google Calendar (not from Gmail, which is doable too) with GoTasks in my iPhone.
RSS reader: Google Reader. Works great. By the way, Orbvious interest (below) also works with Google Reader and has a customizable hotkey for it.
Mark pages to read later: Orbvious Interest for Google Chrome. Great stuff! It syncs between computers and you can use Pocket in your iPad to view the links.
Maps: Google Maps.
Video conversation: Skype, Google Hangouts. If I’m going to help someone remotely, then I use TeamViewer which is free for non-profit purposes. With it you can move their mouse, which makes things much easier for support issues!
Photos: Picasa. I pay the 5 bucks a year for 20 GB on Picasa Web Albums so all my photos are on the cloud.
Dictionary: die.net
Network visualizer/analyzer: Cytoscape.
Venn diagrams with more than 2 sets.
Setting up your website. I pretty much followed Alyssa’s instructions and got my CSS template for my academic page from FCT. Then I used simple html to modify it.

I pretty much dumped a ton of my bookmarks in this huuuuge post! Well, I hope that it will be useful to someone. At least now I’m happy to have contributed to Hilary’s computing-resources-post drive.

International Year of Statistics: coming up soon!

Thu, 16 Aug 2012 00:00:00 +0000

Did you know that next year will be the International Year of Statistics? Well, you probably didn’t! There is a site organizing the activities and listing the institutions that morally support the celebration. I’m happy to see my current work place listed there but at the same time concerned that the National Autonomous University of Mexico is not in it.

Maybe it’s not that surprising since there is no big Statistics department at UNAM. Googling I found a Department of Probability and Statistics listing 14 faculty members and it seems that it was last updated in 2007. Not looking good! There is a more recent site listing 3 staff members and it seems to be geared towards helping students. I did find a graduate program on Mathematical Sciences specializing in Applied Statistics and a short diploma course on Probability and Statistics. I think that most statisticians at UNAM work in institutes without Statistics in the name. I hope that there are plenty more out there!

There are several other Mexican institutions listed (just look at the participants list and search “Mexico” in your browser). Looking at the list highlights how vastly spread is the field. By that I mean that there are plenty of organizations across the world that teach it, and use it. I’m also happy to see that most of the activities (which are almost exclusively conferences) are not in the US. I’m sure that by 2013 there will be plenty more activities listed and hopefully some public outreach events.

05/9/12 PHD comic: 'Grad Stereogram'

Thu, 10 May 2012 00:00:00 +0000

05/9/12 PHD comic: ‘Grad Stereogram’

Is there really something hidden in the pic? I couldn’t see it :P I guessed that there shouldn’t be anything from the text, but just in case I checked.

Pi

Sat, 28 Apr 2012 00:00:00 +0000

More here.

(Saw it from a fb friend).

PhD Comics about beers!

Thu, 08 Mar 2012 00:00:00 +0000

I love the “I went to have a few beers with my friends”!

Pollination video

Thu, 01 Mar 2012 00:00:00 +0000

Great video! Thank you aunt for sending it to me =)

P-values and Statistics phylosophy

Mon, 30 Jan 2012 00:00:00 +0000

I’m in the process of catching up with all the posts from SimplyStatistics that I didn’t read during the break. Doing so I found a very interesting post on p-values (more below)

simplystatistics:

This post written by Jeff Leek and Rafa Irizarry.

The p-value is the most widely-known statistic. P-values are reported in a large majority of scientific publications that measure and report data. R.A. Fisher is widely credited with inventing the p-value. If he was cited every time a p-value was reported his paper would have, at the very least, 3 million citations* - making it the most highly cited paper of all time.

It’s like the p-value is a necessary evil. The post is indeed a great read, but what caught my attention was the following statement:

The advent of new measurement technology has shifted much of science from hypothesis driven to discovery driven making the existing multiple testing machinery useful.

I’m taking a class with Chuck Rohde on probability with a strong philosophical component. That lead me to ask him recently how he would try to convince a biologist that he should believe/trust statistics. To explain myself a bit more, I asked the question as I’ve been in environments where people follow their intuition on what is significant or not and disregard whatever statistics says on the same data. So in the past I’ve been in discussion where I’m the one trying to convince others that statistics say something is not significantly different.

I don’t think that I communicated my question correctly as Rohde’s answer is that whenever he says a +- on count data we (as statisticians) are not doing something right! He also reflected on the past as at some time people were quite into mathematical biology, a field that has either died out or transformed into new fields (systems biology comes to mind). Will the same happen to genomics? *Sound of a coin flipping in the air* I don’t have a clue!

In a sense, the post at SimplyStatistics approaches the same issue I was trying to ask Rohde:

Why not explain to our collaborator that the observation they thought was so convincing can easily happen by chance in a setting that is uninteresting. […] In general, we find p-value to be superior to our collaborators intuition of what patterns are statistically interesting and which ones are not.

To end the post, I invite you to read the paper A Brief History of the Hypothesis by David Glass and Ned Hall from back in 2008. It’s one of my all time favorites. I’ve read it a few times and while my memory distorts what they really wrote, in my mind it’s a great summary with a message: it’s time to abandon hypothesis-driven science and continue with question-driven science involving building models from data and not from our hypothesis.

Here are a few quotes from their paper:

We propose that building hypotheses should be abandoned in favor of posing a straightforward question of a system and then receiving an answer, using that answer to model reality, and then testing the reproducibility and predictive power of the model, modifying it as necessary.

They end their paper with:

Thus, although a hypothesis might have been thought to be necessary in the past, it no longer seems to be so. It is better to see science as a quest for good questions to try to answer, rather than a quest for bold hypotheses to try to refute.

Do you agree?

Correctly synching AddThis and Google Analytics social stats

Fri, 27 Jan 2012 00:00:00 +0000

As a minor addition to my previous posts about setting up a blog, I want to detail a bit more how to synchronize your AddThis social sharing statistics (AddThis Analytics) with Google Analytics.

The main help file explaining how to do so has been a bit confusing for me. Though, in part that’s because I didn’t read it completely. Plus there were a few things I didn’t know.

First, Google Analytics (GA) has two ways in which you can add it to your site: synchronous and asynchronous. There are a few posts around, like this one, explaining the difference. Basically, the asynchronous mode is supposed to give your site faster load times (negligible normally) and most importantly more complete data in cases where people close your site before it finishes loading.

I prefer the asynchronous mode, and the recommendation is to include the script tage before the end of the head tag. In a few cases in Tumblr’s free blog themes, this script is included at the bottom of the html code. So, you might be interested in changing this otherwise the great advantage of the asynchronous mode is lost: the page has to load entirely to report stuff to GA.

Once you do so, note how the AddThis manual has a note that the AddThis script has to go after the GA script. I didn’t read this the first time and I was quite puzzled when no social events were showing up in the GA report.

Next, compared to ShareThis, AddThis social events will be shown in your GA report as “facebook”, “twitter”, etc instead of “ShareThis: facebook” (or something like that). That’s only true if you included the “data_ga_social : true” part in the code. Otherwise GA will only shown under the “Social” tab the Google +1 clicks.

In short: use the GA asynchronous code before the end of the head tag. Add the AddThis script tags below it. Happy social sync :)

I have to say that so far I like the AddThis Analytics way more than Google Analytics for the social stats. Part of it is the “viral lift” percent that AddThis shows. The other part is that it seems to contain more data. Though it might be because I just fixed the sync.

You might want to read the:

Beware of mis-assembled genomes: still valid today!

Thu, 26 Jan 2012 00:00:00 +0000

I’ve been recently impressed by Steven Salzberg talk as you might have noticed, and browing his home page I stumbled upon his opinion piece (also by James Yorke): Beware of mis-assembled genomes.

It’s a short note published in 2005, but damn, can anyone deny that it fits perfectly for today’s state of the art in the de novo genome assembly field? I bet no one will. For instance, it’s a solid statement to say:

> The source of most mis-assemblies is, as it has always been, repeats.

He didn’t add a “will always be” or “will be for at least 7 years more” now that we are in 2012, but it feels like this will be the case until we can get accurate (and cheap) reads that span even the longest repeat. Well, maybe we don’t need such huge reads as people have been able to find large genome duplications.

And as I said in my previous post, I’m still surprised by how careless the human genome assembly was carried out as they didn’t track their own steps. I was hoping that wasn’t the case, but clearly it is:

> Indeed, many of the original assemblies of parts of the human genome were done in the mid- and late-1990s, and are now lost.

I’m also impressed by how accurate Steven and James’ prediction was when they forsaw that people were going to be misled in judging assembly quality by contigs size without taking into account mis-assemblies.

They also called upon the bioinformatics community to take action in evaluating genome assemblies. Due to the amount of data nowadays, it feels like a inhuman (well, incluster as infeasible by high power clusters :P, well, incomputable is the correct term) task. But with some funding, I bet Salzberg and colleagues could find a way to do so. At least partially. Yet, as with anything, you need motivation, and I’m note sure they are motivated to clean up the mess.

Extensive comments and review about the recent bake-off of de novo genome assemblers "GAGE"

Wed, 25 Jan 2012 00:00:00 +0000

During this week’s Genomics seminar at the Genome Cafe in the Biostats department, Steven Salzberg gave a talk on his team new published paper: GAGE: A critical evaluation of genome assemblies and assembly algorithms. I worked on a few assembly projects during my time at Winter Genomics, but that was not the main reason why I was immediately submerged into his talk. I think that it was due to his bold comments since comparing genome assemblers is a, hmm…, delicate issue. I really like the confidence he has on his work and the way he projects it when he talks. It might be too preachy for some, but I like it. Plus it helps that I completely agreed on two key points that differentiate GAGE from it’s competitors: dnGASP and Assemblathon.

First, GAGE uses real data sets instead of simulated ones. I know that some might argue that a given data set can have specific properties that are not general or that it’s biased to a certain assembler. It also feels a bit funny, because I started out assembling simulated data too. It certainly had its uses as I learnt a lot. But once you encounter a real data set you learn how complicated things can be, and it can be quite messy as no one gives you perfectly clean data. I haven’t read much about GAGE’s competitors, but regardless of how they simulate their data, I completely agree with Steven that GAGE has the advantage by using four real data sets. Plus, they were quite sensible when choosing the four data sets as they are Illumina data (the most common) with frequently used read sizes and library types. Note that even the bacterial genome have more than one replicon.

The second key point is that the GAGE team made public all the data and assembly recipes available through their official site (which has a great summary in from of a FAQ explaining the project and key differences). They have certainly made an effort to guarantee the reproducibility of their results, which is hard to do and hasn’t been done before. It’s a sad feeling that it took so long for someone to focus on reproducibility. So it feels wrong that they have to stress out how unique this feature is on their paper, but they definitely had to. Hm… can anyone reproduce the human genome assembly? I’m not talking about someone reading the paper and doing it on their institution computers, but someone from the author team. I hope the changelog is saved at least in some kind of repository.

Another important difference between GAGE and say Assemblaton, is that the for GAGE an in-house team ran the assemblers instead of asking the authors of each program to fine tune their results. If you had asked me a year ago, I would surely had supported the idea of asking the authors to run their programs. After all, even if you read all the documentation it’s the authors who know the best tricks on how to use their assemblers (or should be very good users). Yet, I can see the point that in reality it’s not the authors who run their code for each application. It’s a person or team of bioinformaticians (or a biologist struggling to death with UNIX) that has read the manual & papers (hopefully) from a few tools and decided which is his favorite one. During this process they probably ran a few of the assemblers with a small parameter scan and compared the results. The GAGE pipeline is very similar and hence feels much real. They obviously did this process in a more rigorous way and made sure the conditions allowed comparing the assemblers.

One of the steps common to all of their recipes was to run Quake: quality-aware detection and correction of sequencing errors. I didn’t know about this specific tool before, but I did know about the idea. Basically, you plot the distribution of the k-mers multiplicity from your data and do something to those that are possible errors (those k-mers that are unique or have very low multiplicity compared to the expected value); most commonly you try to correct them and if you can’t, you discard them. That’s a very broad explanation and I’m sure that interested readers will download the original paper.

Anyhow, the point is that they cleaned the data sets prior to using any assembler. I couldn’t agree more to the sentence:

High-quality data can produce dramatic differences in the results

Running some kind of preprocessing cleaning tool should help, but you can’t do miracles with crappy data.

This post is getting huge, so I’ll jump to some points I’d like to highlight though it’ll still be very long.

First, I’m amazed by the simple concept that is “N50 corrected”. It does look complicated to calculate, but the idea of splitting contigs when an error (at least a 5 bp indel) is found (they have Sanger-sequence reference genomes for 3 of them) before calculating the N50 size is just great. It’s simple and very effective. By using this statistic and comparing it to the original N50 size you can clearly detect aggressive assemblers that don’t mind adding errors vs highly conservative ones. Then, comparing “N50 corrected” vs the number of errors (as in figure 6) is VERY informative. I just love that figure!

The result is a bit frustrating because the winner is ALLPATHS-LG. Don’t get me wrong, I think that they are doing great work (they introduced new statistics for comparing assemblies, they are exploiting library preparation more than the rest, etc) but it’s simply hard to come by a data set that meets ALLPATHS-LG’ requirements.

Second, no matter which assembler you use, your result is going to contain lots of errors. There is no way around it, it’s a fact! Hm… unless you want tiny contigs (very conservative assemblers) which aren’t really useful. I think that it’ll be important to stress this out to consumers of the technology instead of fueling their wild dreams of high-quality finished (not draft) de novo assemblies.

Third, de novo genome assembly (specially for large genomes) is still a complicated endeavor. You shouldn’t take it lightly!

As you might have noticed, I found this paper (and the talk) to be very stimulating and interesting. And as I forgot to do so during the talk, thank you Salzberg et al. for taking a huge step in the right direction. It’ll surely help those working on the field and that at some point asked themselves:

What will an assembly based on short reads look like?

Which assembly software will produce the best results?

What parameters should be used when running the software?

A couple more notes:

It’s sad, but Velvet didn’t perform as well as I hoped. I had been convinced for a while that it is one of the best assemblers out there. Plus I’m surprised that it couldn’t run on a 256 GB machine for the bumble bee data set.
Take a look at the E-size statistic on the methods section of the paper. It’s interesting that it correlates well with N50 size. At times, I haven’t been too eager to select a best assembly on N50 size as it might not be one of the longest assemblies and I felt like I was wasting data. But it is a very reasonable summary statistic for such a hard problem.
I’m still curious on whether gap-closers like IMAGE: Iterative Mapping and Assembly for Gap Elimination (I don’t know why they didn’t include the name on their paper title >.<) correctly increase scaffold/contig length by re-assembling local border paired-reads.
Check the websites of the competitors. GAGE looks more complete to me and I’m quite surprised dnGASP doesn’t include links to the other sites! The GAGE twitter might be a bit too much (plus it seems abandoned).
Here is a news commentary which talks about the three competitors.
I found it funny when Salzberg declared a winner in the talk, but it’s surely takes some guts to do so and I agree that it had to be done after such a rigorous comparison (or “bake-off” ^_^).
From the advice section in Genomics 2011 (check my long post about it), “Published is better than perfect”. Results from a bake-off like GAGE are going to change quickly since new updates are released quickly, but they are very helpful!!!

Simply Statistics: Why statisticians should join and launch startups

Tue, 24 Jan 2012 00:00:00 +0000

Simply Statistics: Why statisticians should join and launch startups

I really like this post in SimplyStatistics by leekgroup. The topic is worth considering, but what got me hooked was the list of links to startups, which were all new to me. 100plus looks promising, though I don’t have a clue as to what they are going to offer. Sure, I know the general topic, but not much beyond that. As I just flew back to town, Flightcaster and Hipmunk caught my attention. While Flightcaster works faster and has a great FAQ, I don’t find it super useful since it’s limited to US domestic flights, which are not the kind I usually jump into. Plus, without automatic updates it seems kind of a pain to use. After all, I don’t want to become a panic-driven person who has to refresh the status of his flight every hour or maybe even every few minutes. Hipmunk on the other hand is slow, but I like their “agony” ranking :) It also looks easy to find cheap days to fly though I’ll mostly cross-check with Expedia when I need it.

Anyhow, to read more on the title subject check the original post at simplystatistics:

> The tough economic times we live in, and the potential for big paydays, have made entrepreneurship cool. From the venture capitalist-in-chief, to the javascript coding mayor of New York, everyone is on board. No surprise there, successful startups lead to job creation which can have a…

Commenting: Genomics in 2011

Mon, 23 Jan 2012 00:00:00 +0000

Today is my first day of classes and Kasper couldn’t have had a better timing to share the link to Genomics in 2011: challenges and opportunities. There, the Editorial Board members of Genome Biology gave their opinion on: important 2011 papers, influential people for their careers, advice to young scientists, top challenges in their field, and unlimited money projects. I felt identified several times and as:

I think one of the most important skills in research is the ability to communicate ideas. […] practicing both writing skills and oral presentation skills.

here I am sharing my view on the paper.

Overall the paper is a good read and I enjoyed most of it, so I highly recommend reading it.

The top papers section is quite broad and it felt to me like a huge abstract on many papers. Now I want to read the paper on “phenologs”, a concept that seems very simple and natural, and has helped understand a bit more what genes do (functional genomics).

The parts about influential people and unlimited money projects are not so interesting. I felt that it was too personal and I didn’t feel that it related to me much. Regardless, I’d like to highlight some parts of the text:

> * […] showed me the fun of methodology development in computational biology > * […] emphasized establishing a solid foundation in computer science and statistics but also developed our skills to identify relevant and impactful biological questions > * His humble upbringing, ability to admit when he was wrong, and persistence for what was right has continued to inspire me.

The advice section was very appealing to me and I felt reassured as some of the advice I gave even younger scientists (check my talk to LCG students) popped up. I surely learnt a few things too =)

> * Surrounding yourself with people who are smarter than you also helps you raise your game. > * Published is better than perfect. > * Importance of communication > * you need to appreciate the biology as much as, if not more than, the statistical method or computer science > * The success of our field is ultimately measured by the impact we can have on our understanding of biology > * I would say it’s the ability to understand both the experimental and the computational sciences. […] I would learn the key principles in computational biology or at the very least the linguistics, and vice versa.

I hadn’t taken consciousness of the fact that the amount of computing you can do (per dollar) is doubling every year while the size of genomic databases is going up by a factor of 10. I knew something was off and that we are getting more data, but I didn’t know the ratio. Plus, I have to agree with the comment that cloud computing is not going to be the solution though I’m not sure if:

The only solution is to discover fundamentally better algorithms for processing these databases.

Sure, we need better algorithms, but I think that it’s fair to ask for cleaner and higher-quality data. This doesn’t mean that the data format has to be simpler, as:

Far too often, enough biological details are abstracted away so that the solution loses its biological relevance.

Sad, but true.

I do agree with:

Another challenge is to educate people about genomics and to tone down the natural hype of the genomics field.

Just as with any discipline, it’s not easy to explain your field to a random person and it’s a harder job when someone exaggerates a set of results.

The argument that RO1 NIH grants are not built for young genomics scientists is interesting and I do hope that it changes soon. Though I don’t have a clue as to how the following is going to happen:

> the last challenge is to transform the academic review system in our institutions

> Understanding the systems-level ecological rules governing microbial community structure<

surely opened my eyes as it was one of my main interests a few years ago (and stills interests me ^^).

> streamlining methods for turning next-generation data into actionable biology

sounds very fancy!! It’s as fancy, and important, as developing new ways to visualize networks that take into account time instead of viewing all the possibilities at once.

To finish off the post, I agree completely with:

>I believe the biggest bottleneck is the bioinformatics and the shortage of researchers in the field. There needs to be a big investment to address this shortage.

Note that I think that there are lots of bioinformatics-converts: people who recently joined the field to fill in the gap in their lab. As:

>In my opinion, too many computational biology researchers are working in isolation on marginally relevant problems or making incremental improvements in areas that have already been well-populated by methods that are already adequate.

So, yes, we need more bioinformaticians as most labs have high-throughput experiments (with only one experiment you already need a bioinformatician), but I do hope that a good proportion of them are trained to develop methods and focus on important problems and new areas instead of marginally relevant ones on “old” areas.

That’s why I’m studying a PhD in Biostatistics!

Are you good at programming? You probably practice a lot!

Sat, 10 Dec 2011 00:00:00 +0000

What makes a great programmer? I would have said training, motivation, a good and efficient framework for continuous learning from others such as reading blogs like R-bloggers (if you are an R programmer). Well, on the Occupational Digest blog they commented a paper where they found that the best predictor is programming knowledge. This is acquired through years of practice and makes sense. After all, even if you are a gifted person for something, if you don’t practice and learn from others you won’t get anywhere. So, welcome to school for life! :P

The new visualization package for genome data in Bioconductor: ggbio

Tue, 06 Dec 2011 00:00:00 +0000

It’s been a while since I’ve been waiting for the release of a visualization package in Bioconductor. Back in 2008 I was really impressed by the power of GenomeGraphs and I have used it in multiple occasions. Yet from both the Bioconductor Developer Meeting in Heidelberg 2010 and BioC2011 I’ve been waiting for the release of the visualization tools developed by Michael Lawrence and Tengfei Yin at Genentech.

So, after a long hiatus where I didn’t browse the biocviews in Bioconductor, I found out that Lawrence and Yin released ggbio and biovizBase (it’s more of an infrastructure package for ggbio) . I haven’t really had the time to play around with them, but it’s definitely worth exploring both of their vignette files: ggbio, biovizBase. I also think that they’ll fit very well in Bioconductor because quite a few of their examples involved the gamma of objects the BioC team has released for high-throughput sequencing (HTS) data. Meaning that they work well with objects from IRanges and GenomicRanges. Also, some of the examples use BAM files which are common nowadays in any HTS analysis pipeline. As a plus, ggbio uses ggplot2, which definitely makes clear nice plots.

I expect ggbio to replace GenomeGraphs soon (although I love using it), but I’m also kind of disappointed that I didn’t see any of the cool examples from BioC2011 in ggbio’s vignette file. After all, visnab looked pretty impressive as you can see in this presentation. I don’t know if they decided to rename visnab into ggbio, or maybe they haven’t released visnab yet. Anyhow, give ggbio and biovizBase vignette files a look :)

Introducing Biostatistics to first year LCG students

Sun, 04 Dec 2011 00:00:00 +0000

Around two weeks ago I gave a talk via skype to the first year students from the Undergraduate Program on Genomic Sciences (LCG in Spanish) from the National Autonomous University of Mexico (UNAM in Spanish). The talk was under the context of the Introduction to Bioinformatics Seminar Series whose goal is to familiarize the new students with the bioinformatics world. It used to be a course heavy on exploring database websites, some basic theory, and lots of new concepts and algorithm names. Like, what is BLAST? This year, the course involved several talks from former students (like myself) on their experience and current job (most of us are in graduate school).

In my case, I was invited to talk about Biostatistics and R as I was one of the first LCG students to learn and teach R to other students (including PhD students ^^): 12 hour intro to R and Bioconductor, R in an intro to statistics course, a full course on Bioconductor.

I had a lot of fun preparing my talk as I tried to portray the three main currents in Biostatistics in a way that would be understandable, basic concepts such as a P-value, some basic R code (the students knew the super basics only), and doing so in a way that I would also pass how I see things. A key part for any bioinformatician, as I see it, is to have a good basic toolset. That’s why I tried to pass on many tips to these young students. Also, I do like to go back to the philosophy of science and whether we need hypothesis nowadays or just models based on the data. I definitely had to cover the topic of communication as I strongly believe that any researcher has to be able to communicate with a biostastician if they want their help in analyzing their data. Also, a biostatistician is not a stastistician, meaning that they have to understand the underlying biological question. I tried emphasizing this point with the students and I attempted to motivate them to take a basic stastistics and probability course (2 preferably). After all, biostatistics is key nowadays in science since biology has gone high-throughput.

Giving the talk through skype was definitely a new experience for me. The best is to be in the classroom as you miss a lot of the interaction with the students. For instance, I didn’t always know who as asking the question and frequently I had to ask them to repeat it louder. Plus, I chose to share my desktop with them instead of having them watch at my face all the time. I guess it wasn’t easy for them to just watch a screen with a background voice for 2 hours :P But well, I hoped they liked it as much as I liked preparing it.

If you are interested in the talk, I uploaded the PDF file, the associated R file and the master Rnw file to Google Docs. The Rnw file can be useful if you are learning how to write Beamer presentations using Sweave and LaTeX.

Finally, I was asked to prepare two short questions so that the teachers can evaluate the students. As a bonus for any of them reading my blog ^^ (I did portray blogs as an excellent way to update yourself) here they are:

Argue why you need to learn (at least basic) Biostatistics and why it’s useful nowadays to analyze biological data.
Based on the data from your recent experiment, you construct a 95% confidence interval (frequentist approach) for the mean of your variable of interest. What is the probability that the true mean of the distribution is contained in the confidence interval.
Bonus: list 5 basic R plotting functions

Answers:

You need to learn the basics (at least) of Biostatistics in order to be able to communicate with your biostatistician collaborator, and to analyze your own data specially by performing exploratory data analyses (EDA). Also, biological experiments nowadays generate a lot of data as biology has gone high-throughput. Biostatistics is specially useful in order to analyze all this data.
0 or 1 and you cannot know which is the case for your specific dataset.
I mentioned: plot, hist, boxplot, qqplot, qqnorm, lines, points, abline, legend

Make it easy for your visitors to suscribe with a burnt RSS feed

Wed, 30 Nov 2011 00:00:00 +0000

Recently I talked about how to set up your own blog and now I’m going to continue by emphasizing the importance of an RSS feed. What is it? Basically it is a unique format in which content from frequently updated sites (like a blog) is syndicated. Web syndication allows you to share the content of your site (blog) to many different sites automatically. This is specially useful for people who are interested in following your blog. Sure, you can let them type your blog’s site each time or simply allow them to receive updates on their email, feed reader (I recommend Google Reader), etc. After all, you don’t want your followers to have to remember to visit your site every time if there is an easier option out there. Sure, you already made your blog social but that was meant to facilitate comments and sharing of your blog’s content by it’s visitors. While publishing your blog to Twitter will get initial visitors into the blog, an RSS feed will complete the job when they suscribe to it.

Blogging platforms will normally create an RSS feed for your blog. Yet, this is a plain XML file which is just a pain to read. However, the good news is that you can burn your feed which makes it pretty. That’s where FeedBurner comes into play.

To use it, you simply have to give it your RSS feed XML url and choose a name. For example, the RSS feed XML url for Fellgernon Bit is http://fellgernon.tumblr.com/rss which is rather ugly, but looks much better with FeedBurner at http://feeds.feedburner.com/FellgernonBit The content is readable and FeedBurner allows your visitors to suscribe using a wide variety of tools with a few clicks.

This is not the end, as you can create lots of interesting gadgets through FeedBurner. For example, you can add a link with the number of suscribers (as counted through FeedBurner):

To do so, enter the FeedBurner page for your blog then click on the “Publicize” tab and go to “FeedCount”. There you’ll have several options (like choosing the color, type, etc) and you’ll get a piece of html code that creates the link:

Now simply paste it in the html file of your blog wherever you want it to appear.

FeedBurner offers other options, statistics, etc but I think that one of the most important one is BuzzBoost (again under the Publicize tab). This allows you to show the most recent entries of your blog in other sites you own. Friends might also want to link your blog. In my case, I’m using it show the most recent posts from salmoblog.org as I want to publicize it, plus I’m part of the contributors there. As with FeedCount, BuzzBoost has several options and will give you an html code. Here is the code for Fellgernon Bit:

In the end, if you make your blog it’s worth the extra minutes to configure your RSS feed and make it easy for your visitors to suscribe to it. Plus, you can use FeedBurner to make it nice looking and add some cool gadgets if you want.

Setting up your blog

Sun, 20 Nov 2011 00:00:00 +0000

Now that I’ve spent time re-doing Fellgernon Bit, I thought it’d be a good time my experience on setting up a socially-connected blog.

First of all, you need to choose a blog platform. There are some around like Blogger and WordPress that are widely used and were some of the first platforms. In my case though, I really like how easy http://www.tumblr.com/ is to use.

Once you register and have the default blog, go to the customize menu from the Tumblr dashboard and browse the Free Themes. I ended up choosing the Simple Things theme because I liked the way it looks and it already included some things I wanted like the Google+ button, Facebook and Twitter sharing. This theme also has several easy options so you can easily link your blog to Google Analytics.

One of the very nice gadgets this theme has (as other Tumblr themes) is an easy to customize Disqus account. What is this? Basically, it’s a system for comments. If the visitor registers at Disqus they can choose to share their comments via the main social networks, which is something that I liked. An alternative is the Facebook Social Plugin, but I don’t like because only facebook-users can comment. I know this sounds strange to many of you, but I don’t think that everyone has a fb account and with newer alternatives I believe they’ll lose some of their current userbase. Anyhow, with Disqus visitors can use their Twitter, Fb, Google, OpenId, … accounts to identify themselves and they can easily have their comments appear in the social networks.

One of the cool things Disqus offers is an easy to install widget where visitors can see the top commenters, recent comments, and most discussed threads (or only of the three). To set it up, go to the Admin window and then go to the Tools tab and click on Code.

As you might have noticed, you will have to modify the html file from your theme if you want to connect it and/or add widgets. Don’t be afraid and just follow the simple instructions all these widgets give you.

Once you have your comment system set up, you might be interested in adding more social networks for your visitors to share your posts on. That’s where ShareThis becomes quite handy. Note that ShareThis allows you to add Twitter, Fb, Google+ so even if the theme you chose doesn’t come socially-connected you can do it through ShareThis. In my case, I only added the ShareThis button which pretty much allows visitors to use the whole system of social networks.

This theme (and other Tumblr themes) allow you to link the RSS feed of your blog with FeedBurner which definitely makes your RSS feed look nicer and easy to subscribe to with tools like Google Reader.

Another cool widget to add to your blog is a Tag Cloud. This is probably the easiest widget to add and this excellent post describes all the steps you will want to follow. The only difficult part can be choosing where to add the code in the html file. In my case, I simply looked for the RSS tag and added it below. The div html tags can be useful when you are adding widgets.

To top your blog off, a must-have widget is LinkWithin. Basically, it adds 3-5 links to older posts from your blog which definitely helps keep your older posts in the loop. Otherwise they are simply lost in the eternity of the archive.

Finally, don’t forget to change the settings of your blog from the Tumblr dashboard. You can make so every new post is automatically posted on your Twitter and Fb accounts (I hope they add Google+ soon).

This might all seem like a lot of work, but it’s much quicker to do if you know what you are looking for. And I hope that it’ll be useful to you.

Happy social-blogging!

Posts | L. Collado-Torres

Initial impressions from testing Posit Connect

Our use case

Use case summary

Deployment options

Conclusions

Teaching omics methods to LCG-UNAM students

RNA-seq

scRNA-seq

Spatially resolved transcriptomics

Data Analysis

Miscellaneous

Genomics scientists from and in Mexico community building

LCG(EJ)-UNAM network

Emails

Facebook

Slack

X/Twitter and now Bluesky

Bluesky has taken off

Genomics scientists trained/working in Mexico 🇲🇽

Getting people to join Bluesky

HumanPilot: first spatially-resolved transcriptomics study using Visium

What is a fruit? Banana, orange?

Measuring gene activity

Human brain

Tons of manual work to reproduce what we knew!

Finding and validating new DLPFC layer marker genes

What about disorders? How does your HumanPilot study help us understand them?

Fueling the development of methods, and benefiting from it

What followed

Acknowledgments

References

How to reduce the size of a large GitHub repo

Our story

Steps by steps

Get your workspace ready

Diagnose

Remove dirty files

Editing the git history

Update GitHub

Updating HPC files

Finishing step

Summary

Acknowledgments

References

Reproducibility

CZI Lat Am Meeting 2023: some questions I have before the meeting

I'm disappointed by the reaction against the Tokyo 2020 Mexican Women Softball team

Acknowledgments

References

Reproducibility

Cleaning up my R packages and config files

Config files

Wrapping up

Acknowledgments

References

Reproducibility

The start of a new phase for me

Acknowledgments

References

Reproducibility

You just committed a large file and can't push to GitHub

Recurrent problem: you just commited a large file and can’t push to GitHub

Use case story: the issue

Use case story: the solution

Misc notes

Acknowledgments

References

Reproducibility

Research Scientist: an academic career launch pad

What will I do as a Research Scientist at LIBD?

Details

Final decision

Acknowledgments

References

Reproducibility

Diving together into the unknown world of spatial transcriptomics

Spatial transcriptomics

Our project’s history

spatialLIBD

What about disorders? How does your `HumanPilot` study help us understand them?

Editing the `git` history

Update `GitHub`

`spatialLIBD`

`clusterProfiler` installation