Getting Genetics Done

Repost: ctrlvee: Extract external R code and insert inline

2026-05-22T08:34:48.740-05:00

Reposted from the original at https://blog.stephenturner.us/p/ctrlvee-extract-external-r-code-insert-inline-positron-rstudio-addin.

Ever find yourself looking through a pkgdown page or a Quarto book, copying and pasting code chunks from your browser into your IDE? I do, and it’s a minor annoyance.1

My friend and colleague VP Nagraj published a new R package called ctrlvee that makes this a lot easier.

CRAN: https://cran.r-project.org/package=ctrlvee
GitHub: https://github.com/vpnagraj/ctrlvee

It does one thing. Put your cursor anywhere in an R script in Positron or RStudio, call the add-in, provide a URL, and a few milliseconds later you’ll have all the code from that page in your editor, separated by chunk boundaries (along with some metadata and a note to check the license!).

The package README provides a demonstration using the “Data Validation and QA” chapter of my Data Science Team Training book (dstt.stephenturner.us).

Install the package: install.packages("ctrlvee")
Run the add-in. In Positron you’ll open the command palette, search for Run RStudio Addin, then extract external R code and insert inline. You’ll get a modal asking you for a URL.
Paste one in. E.g., https://dstt.stephenturner.us/validation.html
The R code from the website appears in your editor 🚀

Here’s a demo.

Here’s what the extracted/inserted code looks like, from this source.

# -----------------------------------------------------------------
# Chunks fetched by ctrlvee from: https://dstt.stephenturner.us/validation.html
# Strategy: Rendered HTML page
# Date: 2026-05-16 05:14:44
# Chunks: 8
# NOTE: Check the source license before reusing this code.
# -----------------------------------------------------------------

flu <- data.frame(
    week = c(1, 2, 3, 4, 4),
    county = c("Fairfax", "Arlington", NA, "Loudoun", "Loudoun"),
    disease = c("Flu", "Flu", "Flu", "Flu", "Flu"),
    cases = c(23, 41, 18, -5, 12),
    rate = c(2.1, 3.8, 1.6, NA, 1.1)
)

flu

# ---- chunk boundary ----

if (any(flu$cases < 0, na.rm = TRUE)) {
    stop("Negative case counts detected. Inspect raw data before proceeding.")
}

# ---- chunk boundary ----

stopifnot(
    "Negative case counts" = all(flu$cases >= 0, na.rm = TRUE),
    "Missing county values" = !anyNA(flu$county),
    "Duplicate records" = !anyDuplicated(flu[, c("week", "county")])
)

# ---- chunk boundary ----

install.packages("pointblank")

# ---- chunk boundary ----

library(pointblank)

agent <- create_agent(tbl = flu, label = "Weekly flu surveillance") |>
    col_vals_gte(
        columns = cases,
        value = 0,
        label = "Case counts must be non-negative"
    ) |>
    col_vals_not_null(
        columns = c(week, county),
        label = "Week and county cannot be missing"
    ) |>
    rows_distinct(
        columns = c(week, county),
        label = "No duplicate week/county records"
    ) |>
    interrogate()

agent

# ---- chunk boundary ----

create_agent(tbl = flu, label = "Weekly flu surveillance — extended") |>
    col_is_numeric(
        columns = c(cases, rate),
        label = "Case count and rate must be numeric"
    ) |>
    col_vals_in_set(
        columns = disease,
        set = c("Flu", "COVID-19", "RSV"),
        label = "Disease must be from the approved list"
    ) |>
    col_vals_between(
        columns = week,
        left = 1,
        right = 52,
        label = "Week must be between 1 and 52"
    ) |>
    col_vals_gte(
        columns = rate,
        value = 0,
        na_pass = TRUE,
        label = "Rate must be non-negative (NAs allowed)"
    ) |>
    interrogate()

# ---- chunk boundary ----

if (!all_passed(agent)) {
    stop("Data validation failed. Review the agent report before proceeding.")
}

# ---- chunk boundary ----

library(readr)
library(pointblank)

flu <- read_csv("data/flu-2024.csv")

# Validate immediately after reading
agent <- create_agent(tbl = flu, label = "flu-2024 validation") |>
    col_vals_gte(columns = cases, value = 0, label = "No negative counts") |>
    col_vals_not_null(columns = c(week, county), label = "No missing keys") |>
    rows_distinct(columns = c(week, county), label = "No duplicate records") |>
    interrogate()

if (!all_passed(agent)) {
    stop("Validation failed — see agent report above.")
}

Repost: OpenAI Codex App: Vibe-updating my old qqman R package to ggplot2 with plan+execute.

2026-02-04T04:18:00.002-06:00

Reposted from the original at https://blog.stephenturner.us/p/openai-codex-app-qqman.

Earlier this week I reposted a Bluesky post from Ethan Mollick. I’m on the same page here — there are problems for sure, but these tools have irreversibly changed the practice of software development (and IMHO, for the better on balance).

Claude Code has gotten everyone excited recently (including me), even going so far as to make it into the popular press (see the Atlantic article on CC). Codex has been around for a while, but this looks like OpenAI’s attempt to challenge CC’s position.

The new Codex desktop app

This week OpenAI launched a dedicated Codex desktop app positioned as a focused UI for running multiple agents in parallel, keeping changes isolated via built-in worktrees, and extending behavior with skills and scheduled automations. Like the demo I highlighted in my previous post on Claude Code, Codex uses worktrees to isolate tasks/PRs as primitives for conflict resolution, has a plan mode (/plan) to force upfront decomposition and questions, can use skills as reusable bundles that can connect to external services, and has automations for recurring background jobs.

Download it at openai.com/codex. It’s free to use for some unspecified time. Or works with your Plus/Enterprise/Edu/whatever account.

Demo: updating qqman to use ggplot2

My most highly cited paper is about some R code I wrote 15 years ago, and later turned into a package, for creating Manhattan Plots from GWAS data. The paper in JOSS has been cited ~1000 times, and the preprint another ~2100 times.

Turner, S. D. (2018). qqman: an R package for visualizing GWAS results using Q-Q and manhattan plots. Journal of Open Source Software, 3(25), 731. https://doi.org/10.21105/joss.00731.

I wrote the original code using ggplot2, then refactored everything to use base R, because at the time, at least with how I wrote the package, plotting millions of points with ggplot2 as agonizingly slow.

I’ve gotten lots of feature requests for qqman, and I don’t have time to respond to them unfortunately. Many of these feature requests would be easy to implement if the package did in fact use ggplot2, since it’s been so long since I’ve done anything in base R that I’ve forgotten how to do pretty much anything.

So I fired up the Codex app, put it into “plan” mode, and asked for help. Along the way it asks me questions about my preferences.

Here’s a screenshot of the plan (truncated).

#YOLO, implement the plan.

When I rebuilt the package, there were a few of those “no visible global binding for global variable…” notes that I had to fix, but otherwise everything worked. Everything now uses ggplot2. The function returns a ggplot object. Column names are no longer as strict. It’s fast. It uses ggrepel to label points so that they don’t overlap. Documentation was updated. The README was updated. The vignettes were updated. It went off and did this in about 5 minutes.

Here’s one of the plots that comes out of this.

I’m not going to update the package, because I still don’t have time to maintain it or respond to feature requests. But that shouldn’t stop you if there’s something you want to do with qqman or any other open-source R/Python package that the package doesn’t currently do.

Repost: Construct objects with idiomatic R code

2025-10-02T03:54:00.002-05:00

Reposted from the original at https://blog.stephenturner.us/p/construct-objects-with-idiomatic-r-code

---

Today I discovered the constructive package and the construct() function for creating R objects with idiomatic R code to make human-readable reproducible examples.

CRAN: https://cran.r-project.org/package=constructive
Source: https://github.com/cynkra/constructive/
Docs & vignettes: https://cynkra.github.io/constructive/

Imagine you want to create a reproducible example and you need to create an object you can share somewhere like StackOverflow. Here let’s take the starwars data that ships with dplyr, and get just the first few rows, and only a few select columns.

library(dplyr)

swpartial <- 
  starwars |> 
  head(4) |> 
  select(name, species, films)

swpartial

This partial dataset has 4 rows, 3 columns, and one of those is a list-column.

# A tibble: 4 × 3
  name           species films    
  <chr>          <chr>   <list>   
1 Luke Skywalker Human   <chr [5]>
2 C-3PO          Droid   <chr [6]>
3 R2-D2          Droid   <chr [7]>
4 Darth Vader    Human   <chr [4]>

The built-in dput() function gives you the code to recreate this object:

dput(swpartial)

But the output is hardly human readable:

structure(list(name = c("Luke Skywalker", "C-3PO", "R2-D2", "Darth Vader"
), species = c("Human", "Droid", "Droid", "Human"), films = list(
    c("A New Hope", "The Empire Strikes Back", "Return of the Jedi", 
    "Revenge of the Sith", "The Force Awakens"), c("A New Hope", 
    "The Empire Strikes Back", "Return of the Jedi", "The Phantom Menace", 
    "Attack of the Clones", "Revenge of the Sith"), c("A New Hope", 
    "The Empire Strikes Back", "Return of the Jedi", "The Phantom Menace", 
    "Attack of the Clones", "Revenge of the Sith", "The Force Awakens"
    ), c("A New Hope", "The Empire Strikes Back", "Return of the Jedi", 
    "Revenge of the Sith"))), row.names = c(NA, -4L), class = c("tbl_df", 
"tbl", "data.frame"))

Alternatively, load the constructive package and use the construct() function:

library(constructive)
construct(swpartial)

The code you’ll get will create an identical object, but it’s much more human readable, and it’s probably what you would actually write out if you were constructing this object manually.

tibble::tibble(
  name = c("Luke Skywalker", "C-3PO", "R2-D2", "Darth Vader"),
  species = c("Human", "Droid", "Droid", "Human"),
  films = list(
    c(
      "A New Hope", 
      "The Empire Strikes Back", 
      "Return of the Jedi",
      "Revenge of the Sith", 
      "The Force Awakens"
    ),
    c(
      "A New Hope",
      "The Empire Strikes Back",
      "Return of the Jedi",
      "The Phantom Menace",
      "Attack of the Clones",
      "Revenge of the Sith"
    ),
    c(
      "A New Hope",
      "The Empire Strikes Back",
      "Return of the Jedi",
      "The Phantom Menace",
      "Attack of the Clones",
      "Revenge of the Sith",
      "The Force Awakens"
    ),
    c(
      "A New Hope",
      "The Empire Strikes Back",
      "Return of the Jedi",
      "Revenge of the Sith"
    )
  )
)

There are lots of other features and use cases. See the documentation for details.

Repost: Make your development environment portable and reproducible

2025-09-09T05:21:00.000-05:00

Reposted from the original at https://blog.stephenturner.us/p/development-environment-portable-reproducible.

You upgrade your old Intel Macbook Pro for a new M4 MBP. You’re setting up a new cloud VM on AWS after migrating away from GCP. You get an account on your institution’s new HPC. You have everything just so in your development environment, and now you have to remember how to set everything up again.

I just started a new position, and I’m doing this right now.

Setting up a reproducible and portable development environment that works seamlessly across different machines and cloud platforms can save you time and headaches. These are a few of the strategies I use1 to quickly reproduce my development environment across machines.

Dotfiles in a GitHub repo
New VM setup script in a GitHub repo
R “verse” package on GitHub
Dev containers in VS Code

Keep your dotfiles in a private GitHub repo

Dotfiles are the hidden configuration files in your home directory. Examples include .vimrc for Vim, .tmux.conf for tmux, or .bashrc for your shell environment. I have a long list of aliases and little bash functions in a .aliases.sh file that my .bashrc sources. I also have a .dircolors, global .gitignore, a .gitconfig, and a minimal .Rprofile.

Keeping these files in a GitHub repository makes it easy to quickly reproduce your development environment on another machine. If you search GitHub for “dotfiles” or look at the awesome-dotfiles repo, you’ll see many people keep their dotfiles in a public repo. I use a private repo, because I’m too scared I might accidentally commit secrets, such as API tokens in my .Renviron or PyPI credentials in .pypirc.

Whenever you get a new machine or VM, getting things set up is easy:

# Your private dotfiles repo
git clone https://github.com/<yourusername>/dotfiles
cd ~/dotfiles

# A script to symlink things to your home
./install.sh

Keep a fresh cloud VM setup script

I started playing with computers in the 1990s. I’ve experienced enough hard drive failures, random BSODs, and other critical failures, that I treat my computer as if it could spontaneously combust at any moment and I could immediately lose all of my unsaved, un-backed-up work at any moment. I treat my cloud VMs the same way, as if they’re disposable (many times they are disposable, by design).

Imagine you launch a new cloud VM starting from a clean Ubuntu image. Now you need all the tools you use every day on this machine - vim, tmux, RStudio, conda, Docker, gcloud/gsutil, etc. Additionally, while I use conda to create virtual environments for installing tools for specific tasks, there are some domain-specific tools I use so often every day for exploratory analysis that I actually prefer having a local installation on the machine — things like bedtools, seqtk, samtools, bcftools, fastp, Nextflow, and a few others — instead of having to load a conda environment or use Docker every time I want to do something simple.

I keep a script on GitHub that will install all the software I need on a fresh VM. Here’s an example setup script I use as a GitHub gist.

I know this isn’t completely Reproducible™ in the sense that a Docker container might be, because I’m not controlling the version of every tool and library I’m installing, but it’s good enough to get me up and running for development and interactive data analysis and exploration.

R: Custom “verse” package on GitHub

The tidyverse is probably the best known meta-package that installs lots of other packages for data science. Take a look at the tidyverse package DESCRIPTION file. When you run install.packages("tidyverse"), it will install all the packages listed in the Imports field, including dplyr, tidyr, purrr, ggplot2, and others.

You can use this pattern to create your own “verse” package that installs all your favorite packages. This is helpful for setting up a new machine, or re-installing all the R packages you use whenever you upgrade to a new major version of R.

Take a look at my Tverse package on GitHub at github.com/stephenturner/Tverse, specifically at the DESCRIPTION file. In the Imports field I include all the packages I know I’ll use routinely. Note that this also includes several Bioconductor packages (which requires including the biocViews: directive in the DESCRIPTION), as well as one of my favorite packages, breakerofchains, that is only available from GitHub (requiring the Remotes: entry).

Once this package is pushed to GitHub I can easily install all those packages and their dependencies:

devtools::install("stephenturner/Tverse")

Dev containers in VS Code

Development containers (dev containers) allow you to create and use consistent development environments using Docker containers. It allows you to open any folder inside (or mounted into) a container and take advantage of Visual Studio Code's full feature set. This is particularly useful when working with teams or switching between projects with different dependencies.

The dev container docs and tutorial are both good places to start. You’ll need to have Docker running, and install the Dev Containers VS Code extension.

From Microsoft’s documentation:

Workspace files are mounted from the local file system or copied or cloned into the container. Extensions are installed and run inside the container, where they have full access to the tools, platform, and file system. This means that you can seamlessly switch your entire development environment just by connecting to a different container.

From Microsoft’s dev containers documentation.

Using a dev container template

You can use any pre-built dev container templates available on registries like Docker Hub or Microsoft’s container registry. Here’s an example using Rocker with R version 4.4.1, and adds a few extensions to VS Code running in the container. You could also create your own container for development, put that on Docker Hub, then use that image.

{
    "image": "rocker/r-ver:4.4.1",
    "customizations": {
        "vscode": {
            "extensions": [
                "REditorSupport.r",
                "ms-vscode-remote.remote-containers"
            ]
        }
    }
}

Using a custom Dockerfile

You can use a custom Dockerfile to create your dev container. First, create a .devcontainer/ directory in your project with a Dockerfile and a devcontainer.json file. Define your development environment in the Dockerfile (base image, installed packages and configuration). In the JSON replace the image property with build and dockerfile properties:

{
    "build": {
        "dockerfile": "Dockerfile"
    }
}

Start VS Code running the container

After you create your devcontainer.json file (either from a template or completely custom), open the folder in the container using the command palette:

And prove to yourself that your VS Code environment is indeed using the container (I’m using rocker R 4.4.1 here). Running whoami shows I’m root inside the container (not my own username), and I’m indeed running R version 4.4.1.

Vagrant and Ansible are powerful tools for managing development environments and automating configurations. Vagrant allows you to create and configure lightweight, reproducible, and portable virtual environments, while Ansible automates complex system setups and deployments across multiple machines. However, they can be overkill for simple or personal development environments, so I'm focusing on lighter, more straightforward solutions.

Using OpenAI Codex in Positron

2025-09-09T05:19:00.004-05:00

Reposted from the original at https://blog.stephenturner.us/p/codex-positron.

Last month I wrote about agentic coding in Positron using Positron assistant, which uses the Claude API on the back end.

Positron Assistant: GitHub Copilot and Claude-Powered Agentic Coding in R

Stephen Turner

Jul 16

Read full story

Yesterday OpenAI announced a series of updates to Codex, the biggest being an IDE extension to allow you to use Codex in VS Code, Cursor, Windsurf, etc. More details at developers.openai.com/codex. And Codex is available in the Open VSX Registry, meaning you can install it in Positron.

Demo: creating an R package with Codex

I tried doing the same thing here with Codex as I did with Positron Assistant in the previous post. I used usethis::create_package() to give me a basic package skeleton, then I fired up Positron, hit the Codex extension in the side panel, and gave it a simple prompt.

write a simple function in this R package to reverse complement a DNA sequence (i.e. A>T, C>G, G>C, T>A). Document it with Roxygen, and write unit tests with testthat. Do not add any external package dependencies other than testthat.

Then I sat back and watched it work.

As you can see, after running devtools::document() and devtools::test(), my tests failed. I asked Codex to fix those tests. I had to do this twice, and the second time around it’s running those tests locally and diagnosing what’s happening.

The third time around all my tests pass.

And devtools::check() yields no errors, warnings, or notes.

The code is on the same GitHub repo, on the codex branch.

Why Codex instead of Positron Assistant?

I haven’t used either agent enough to know their failure modes, and which might be better in certain circumstances. As of last week, GPT-5 seems to outperform Claude for writing R code, and Codex uses GPT-5 under the hood.

Another factor might be cost. Instead of using API credits, Codex uses your existing ChatGPT Plus, Team, Pro, Edu, or Enterprise subscription. In my post on Positron Assistant I showed that the entire package development experiment (admittedly simple) cost about $0.09 cents. But if you’re relying on this daily and using it for heavier tasks, you might run up a decent bill. If you’re already paying $20/month for ChatGPT Plus, using Codex doesn’t cost you any more.

Finally, there’s the original selling point behind Codex before it was ever available in an IDE: You can wire up Codex to your GitHub account and ask Codex to read, write, and execute code in your repositories to answer questions or draft PRs. I haven’t tried this yet, but you can read more at developers.openai.com/codex/cloud.

Positron Assistant: GitHub Copilot and Claude-Powered Agentic Coding in R

2025-09-09T05:18:00.000-05:00

Reposted from the original at https://blog.stephenturner.us/p/positron-assistant-copilot-chat-agent

I have a little hobby project I’m working on and I wanted to use the opportunity to fully make the switch to Positron from RStudio. I used Positron here and there when it first came out, but now that it’s out of beta and has a more complete feature set (like remote SSH sessions!) I have everything I need to switch and not look back. The most exciting new addition is the new Positron Assistant.

Positron Assistant

I wrote a post last year about AI code completion in Positron. GitHub copilot wouldn’t work in Positron at the time so I tried out Codeium, Tabnine, and Continue.

AI code completion in Positron

Stephen Turner

October 1, 2024

Read full story

Using a third-party plugin is no longer necessary. One of the more exciting new features in Positron is Positron Assistant.1 From the description:

Positron Assistant is an AI client that provides LLM integration within Positron, both for chat and for inline completions. Use Positron Assistant to generate or refactor code, ask questions, get help with debugging, and get suggestions for next steps in your data science projects.

Positron Assistant allows you to use GitHub Copilot for inline code completions, and Anthropic Claude for chat and agent mode. The documentation has instructions for getting this set up so I won’t go into those details. You make a configuration change in Positron, then sign into your GitHub account with OAuth, and put in your Anthropic API key, and you’re off to the races.

Cmd-Shift P to bring up the command pallette in Positron, then search for “Positron Assistant: Configure Language Model Providers.”

Code completion with GitHub Copilot

This isn’t anything new. GitHub Copilot has been available in VSCode and RStudio for years. But it’s nice to have it available in Positron now.

Here’s a demo where I’m starting with a blank R script, and write comments in the code describing what I want, then let Copilot take it away as I just hit the tab key to accept the suggestions. Here I’m asking for a function to reverse complement a DNA sequence. Here’s the code it produced.

Agent mode to create an R package

When Positron first came out I wrote about using it for R package development.

R package development in Positron

Stephen Turner

July 29, 2024

Read full story

I wanted to try out Positron Assistant’s agent mode to see how it works with R packages. Cursor and Claude Code seem to be all the rage on all the tech podcasts, Twitter feeds, and blogs I follow, but I’ve been reluctant to switch IDEs (or in the case of Claude Code, ditching the IDE altogether).

Activate the Assistant in Positron’s sidebar, then select Agent mode.

I started up a fresh R session and ran usethis::create_package() to create a blank package. This just creates the bare minimum (DESCRIPTION, NAMESPACE, etc.) needed for a skeleton R package. Then I activated Positron Assistant in agent mode, asked it to write a function in the package to reverse complement a DNA sequence, document it with Roxygen, and write unit tests with testthat.

It’s fun to sit back and watch the agent work. It scans the directory structure, finds the R version, creates the function, writes the documentation, writes the tests, then presents a model asking me whether I want to run the tests that it just created. It wrote everything in one shot with all tests passing and no errors on devtools::check().

Everything you see here cost $0.09 cents using the Claude 4 Sonnet API.2

The one thing I had to fix was the License field in the DESCRIPTION file with a simple usethis::use_mit_license(). The default for this field came in from usethis::create_package() and was simply boilerplate telling me that I needed to choose a license. Once I fixed this all tests passed, and the package check came out clean with 0 errors, warnings, or notes. I uploaded the package here on GitHub.

View the package code on GitHub

It was honestly pretty mesmerizing to sit back and watch the agent do its thing, inspecting the environment, writing code, docs, and tests.

Obviously this was a simple greenfield example, and I’d be curious to see how the agent handles larger codebases with complex dependencies and newer coding paradigms (like R’s new S7 OOP system) that won’t have good training data from Stack Overflow or elsewhere.

At the time I’m writing this (July 2025) Positron Assistant is still in preview, meaning that features might change by the time you’re reading this. For instance, currently only GitHub Copilot is available for inline code completions, and only Anthropic Claude is available for chat and agent mode. I’m sure both of these will expand in the near future to allow for other model providers (although Claude 4 consistently ranks at the top for R coding capabilities).

So many people I’ve talked to have no issue paying $20/month for ChatGPT Plus or Claude Pro, but are reluctant to buy API credits. I’m not sure how to rationalize this. I think there might be a misunderstanding that it works like AWS, where you put in a credit card and could accidentally rack up a huge bill. It doesn’t work like this. It’s a prepaid service. I put $5 on my account months ago just to experiment around a bit and I still haven’t used it all. You can set rate limits and email notifications on your API keys if you’re worried about spending more than a few pennies trying out something like what you see in this post.

Repost: Tidy RAG in R with ragnar

2025-07-15T17:43:00.004-05:00

Reposted from the original at: https://blog.stephenturner.us/p/tidy-rag-in-r-with-ragnar

Retrieval augmented generation in R using the ragnar package. Demonstration: scraping text from relevant links on a website and using RAG to ask about a university's grant funding.

Note: After I wrote this post last week, the Tidyverse team released ragnar 0.2.0 on July 12. Everything here should still work, but take a look at the release notes to learn about some nice new features that aren’t covered here.

I’ve written a little about retrieval-augmented generation (RAG) here before. First, about GUIs for local LLMs with RAG:

GUIs for Local LLMs with RAG

Stephen Turner

Mar 14

Read full story

…and later on building a little RAG app to chat with a bunch of PDFs in your Zotero library using Open WebUI:

Build a local RAG application with Open WebUI to chat with your Zotero library

Stephen Turner

Apr 5

Read full story

In an oversimplified nutshell: LLMs can't help you with things that are not in their training data or are past their training cutoff date. With RAG, you can provide relevant snippets from those documents as context to the LLM so that its answers are grounded in a collection of known content from a trusted document corpus.

Even more oversimplified: RAG lets you “chat with your documents.”

In this post I’ll demonstrate how to scrape text from a website and implement a RAG workflow in R using a new addition to the tidyverse: ragnar, along with functionality from ellmer to interact with LLM APIs through R.

Demonstration

Python has historically dominated the AI product development space, but with recent additions like ellmer, chores, gander, and mall, R is quickly catching up.

Here I’m going to use the new ragnar package in the tidyverse (source, documentation) to build a little RAG workflow in R that uses the OpenAI API.

I’m going to ingest information from the UVA School of Data Science (SDS) website at datascience.virginia.edu, then ask some questions that won’t have answers in the base model’s training data.

Setup

If you want to follow along you’ll need an OpenAI API key. You can set that up at platform.openai.com. Once you do that, run usethis::edit_r_environ() to add a new OPENAI_API_KEY environment variable, and restart your R session.

In R I’m going to need the ellmer and ragnar packages. Because ragnar isn’t yet on CRAN, I'll have to install it with pak or devtools.

install.packages("ellmer")
pak::pak("tidyverse/ragnar")

Create a vector store

The first thing I want to do is to find all the links to other pages at datascience.virginia.edu, scrape all of that content, and stick it into a DuckDB database. Most of this is modified straight from the ragnar documentation, hence the context chunking still looks like I’m ingesting a book.

library(ragnar)

# Find all links on a page
base_url <- "https://datascience.virginia.edu/"
pages <- ragnar_find_links(base_url)

# Create and connect to a vector store
store_location <- "pairedends.ragnar.duckdb"
store <- ragnar_store_create(
  store_location,
  embed = \(x) ragnar::embed_openai(x),
  overwrite=TRUE
)

# Read each website and chunk it up
for (page in pages) {
  message("ingesting: ", page)
  chunks <- page |>
    ragnar_read(frame_by_tags = c("h1", "h2", "h3")) |>
    ragnar_chunk(boundaries = c("paragraph", "sentence")) |>
    # add context to chunks
    dplyr::mutate(
      text = glue::glue(
        r"---(
        # Excerpt from UVA School of Data Science (SDS) page"
        link: {origin}
        chapter: {h1}
        section: {h2}
        subsection: {h3}
        content: {text}

        )---"
      )
    )
  ragnar_store_insert(store, chunks)
}
# Build the index
ragnar_store_build_index(store)

Retrieval

Now suppose we want to ask questions about research grant funding at the UVA School of Data Science (SDS). First, let’s see what ChatGPT tells us without providing any context at all when we ask it to tell us about SDS’s active grant funding.

I don’t have specific information about “SDS’s active grant funding” because “SDS” could refer to several different organizations or entities (e.g., Social and Decision Sciences departments, Sustainable Development Solutions, Students for a Democratic Society, or a company with the acronym SDS).

Now let’s use the OpenAI API providing context from the vector store we just created that should have information about UVA SDS’s active funding from their active grants listing page.

library(ragnar)
store_location <- "pairedends.ragnar.duckdb"
store <- ragnar_store_connect(store_location, read_only = TRUE)

# Get relevant chunks
text <- "Tell me about some of SDS's active grant funding."
relevant_chunks <- ragnar_retrieve_vss(
  store,
  text,
  top_k = 10
)
relevant_chunks

#'  Register ellmer tool
#' You can register an ellmer tool to let the LLM retrieve chunks.
system_prompt <- stringr::str_squish(
  r"--(
  You are an expert about UVA's School of Data Science.
  You are concise. You always respond by first direct
  quoting material from the SDS website, then adding
  your own additional context and interpertation.
  Always include links to the source materials used.
  )--"
)
chat <- ellmer::chat_openai(
  system_prompt,
  model = "gpt-4.1-mini",
  params = ellmer::params(temperature = .5)
)

# Register a retrieve tool with ellmer
ragnar_register_tool_retrieve(chat, store, top_k = 10)

# Run the query
chat$chat(text)

Now when asking about some of SDS’s active grant funding, we get a better response:

Here are some examples of active grant funding at UVA's School of Data Science (SDS):
Resource and Data Exchange (RDE) has a grant award of $5,000,000 with a performance period from Feb 29, 2024 to Feb 27, 2029, currently 25% completed.
Biomedical Data Science Innovation Labs: An Intensive Research Project Development Program has an award amount of $1,943,352.
Supporting Pre-Service Teachers Mathematical Discourse through Co-Design of Teaching Simulation Tools is sponsored by the US NSF with a total award of $250,000.
Activity Aware Bayesian Deep Learning is sponsored by the US NSF.
Enhancing Security and Privacy of Community Cyberinfrastructures for Collaborative Research has a total award of $882,518 from the US NSF.
Evaluating the Efficacy of Repurposed Medications has an award amount of $86,134.
Accelerating racial and ethnic equity in SCI rehabilitation outcomes has an award amount of $53,170.
The application of data science on financial systems with an emphasis on safety and security has an award amount of $5,000.
These grants cover a range of topics from education, biomedical innovation, security, to social equity in rehabilitation.
Source: UVA SDS Active Grants

API cost and local models

As I’m writing this, the cost for GPT-4.1 mini is ridiculously cheap at $0.40 cents per million input tokens (see more on their API pricing page). The demonstration here cost me $0.01 cent (the text embedding and vector storage cost a fraction of a penny in addition to the input/output completions).

There are plenty of open/local models that support tool use, as well as open/local text embedding models, all of which can be run through Ollama. I tried the same exercise above using Nomic Embed through Ollama for text embedding, and tried several with tool calling abilities, including qwen3, mistral, llama3.1, llama3.2, llama3.3, and the new llama4, and the results were all terrible. I don't know if this was due to the inferiority of the models themselves, or if this was the embedding model that I chose, which incidentally happened to be the most popular embedding model available in Ollama. Just put $1 on your OpenAI API account and get to work and stop worrying about it.

Learning more

This recent webinar from Posit CTO Joe Cheng doesn’t cover RAG at all. In fact, he mentions near the top that RAG should not be your first choice when simply changing a system prompt would be good enough. It’s a good talk and I learned a few nice things along the way.

Repost: The Modern R Stack for Production AI

2025-06-03T04:10:00.002-05:00

Reposted from the original at: https://blog.stephenturner.us/p/r-production-ai

...

Python isn't the only game in town anymore: R can interact with local and cloud LLM APIs, inspect and modify your local R environment and files, implement RAG, computer vision, NLP, evals, & much more.

There was a time in late 2023 to early 2024 when I and probably many others in the R community felt like R was falling woefully behind Python in tooling for development using AI and LLMs. This is no longer the case. The R community, and Posit in particular, have been on an absolute tear bringing new packages online to take advantage of all the capabilities that LLMs provide. Here are a few that I’ve used and others I’m keeping a close eye on as they mature.

ellmer: interact with almost any LLM in R

CRAN: https://cran.r-project.org/package=ellmer
GitHub: https://github.com/tidyverse/ellmer/
Documentation: https://ellmer.tidyverse.org/
Blog post: https://posit.co/blog/announcing-ellmer/

I can't remember when I first started using Ollama to interact with local LLMs like Llama and Gemma, but I first used the ollamar package (CRAN, GitHub, pkgdown) last summer, and wrote a post on using this package to ask Llama3.1 what’s interesting about a set of genes, or to summarize papers on bioRxiv.

Use R to prompt a local LLM with ollamar

Stephen Turner

August 14, 2024

Read full story

Shortly after that, I wrote an R package to summarize bioRxiv preprints with a local LLM using ollamar:

biorecap: an R package for summarizing bioRxiv preprints with a local LLM

Stephen Turner

August 24, 2024

Read full story

In addition to the ollamar package, Johannes Gruber introduced the rollama package (CRAN, GitHub, pkgdown) around the same time, though I haven’t used it myself.

Earlier this year Posit announced ellmer, a new package that allows you to interact with most major LLM providers, not just local models running via Ollama. The ellmer package supports ChatGPT, Claude, AWS Bedrock, Azure OpenAI, DeepSeek, Gemini, Hugging Face, Mistral, Perplexity, and others. It also supports streaming and tool calling. I wrote another post more recently demonstrating how to summarize Bluesky posts on a particular topic using ellmer:

Bluesky conversation analysis with local and frontier LLMs with R/Tidyverse

Stephen Turner

December 30, 2024

Read full story

The ellmer package’s documentation and vignettes are top-notch. Check it out.

chores: automate repetitive tasks

CRAN: https://cran.r-project.org/package=chores
GitHub: https://github.com/simonpcouch/chores/
Documentation: https://simonpcouch.github.io/chores/
Blog post: https://posit.co/blog/introducing-chores/

The chores package connects ellmer to your source editor in RStudio and Positron, providing a library of ergonomic LLM assistants designed to help you complete repetitive, hard-to-automate tasks quickly. These “assistants” let you do things like highlight some code and convert tests to testthat3, document functions with roxygen, or convert error handling to use cli instead of stop or rlang. There’s a nice demonstration screencast on the documentation website.

gander: allow an LLM to talk to your R environment

CRAN: https://cran.r-project.org/package=gander
GitHub: https://github.com/simonpcouch/gander/
Documentation: https://simonpcouch.github.io/gander/
Blog post: https://posit.co/blog/introducing-gander/

The gander package feels kind of like a Copilot but it also knows how to talk to objects in your R environment. It can inspect file contents from elsewhere in the project that you're working on, and it also has context about objects in your environment, like variables, data frames, and functions. There’s a nice demonstration screencast on the documentation website.

btw: describe your R environment to an LLM

CRAN: (Not yet on CRAN)
GitHub: https://github.com/posit-dev/btw/
Documentation: https://posit-dev.github.io/btw/
Blog post: https://posit.co/blog/custom-chat-app/

The btw package is brand new and still in development. You can use it interactively, where it assembles context on your R environment, package documentation, and working directory, copying the results to your clipboard for easy pasting into chat interfaces. It also allows you to wrap methods that can be easily incorporated into ellmer tool calls for describing various kinds of objects in R. I’d recommend reading Posit’s “Teaching chat apps about R packages” blog post.

ragnar: retrieval-augmented generation (RAG) in R

CRAN: https://cloud.r-project.org/package=ragnar
GitHub: https://github.com/tidyverse/ragnar/
Documentation: https://ragnar.tidyverse.org/

The ragnar R package helps implement Retrieval-Augmented Generation (RAG) workflows in R using ellmer to connect to any LLM you wish on the backend. It provides some really handy utility functions for reading files or entire websites into a data frame, converting files to markdown, and finding all links on a webpage to ingest. It helps you chunk text into sections, embed into a vector store (using duckdb by default), and retrieve relevant chunks to provide an LLM with context given a prompt.

I’m working on another post right now with a deeper dive into using ragnar.

vitals: LLM evaluations in R

CRAN: (Not yet on CRAN)
GitHub: https://github.com/tidyverse/vitals/
Documentation: https://vitals.tidyverse.org/

LLM evaluation at R, aimed at ellmer users. It’s an R port of the widely adopted Python framework Inspect. As of this writing, the documentation notes that vitals is highly experimental and much of its documentation is aspirational.

kuzco: computer vision in R

CRAN: (Not yet on CRAN)
GitHub: https://github.com/frankiethull/kuzco
Documentation: (No pkgdown site yet)
Blog post: https://posit.co/blog/kuzco-computer-vision-with-llms-in-r/

The kuzco package is designed as a computer vision assistant, giving local models guidance on classifying images and return structured data. The goal is to standardize outputs for image classification and use LLMs as an alternative option to keras or torch. It currently supports classification, recognition, sentiment, and text extraction.

mall: use an LLM for NLP on your data

CRAN: https://cloud.r-project.org/package=mall
GitHub: https://github.com/mlverse/mall
Documentation: https://mlverse.github.io/mall/

The mall package provides several convenience functions for sentiment analysis, text summarization, text classification, extraction, translation, and verification.

I recently used the mall package to run a quick sentiment analysis of #Rstats posts on Bluesky:

Bluesky conversation analysis with local and frontier LLMs with R/Tidyverse

Stephen Turner

December 30, 2024

Read full story

Functions in the mall package integrate smoothly with piped workflows in dplyr. For example:

reviews |>
  llm_sentiment(review)

Other resources

I think this is just the tip of the iceberg, and I can’t wait to see what else Posit and others in the R community are doing in this space.

Here's a recording from a recent webinar Joe Cheng (Posit CTO) gave on harnessing LLMs for data analysis.

You might also take a look at the recordings from posit::conf(2024) which include a few AI/LLM-focused talks.

Also check out the posit::conf(2025) schedule at https://posit.co/conference/. There’s a workshop on Programming with LLM APIs: A Beginner’s Guide in R and Python, four talks in a session titled LLMs with R and Python, several lightning talks that will likely cover LLMs in R, and four more talks in a session titled Facepalm-driven Development: Learning From AI and Human Errors.

The R community has clearly stepped up. Whether you're building prototypes, shipping production tools, or just exploring what LLMs can do, R is now a real and robust option. I’m excited to see where we go from here.

Find this useful? Buy me a coffee! ☕️

Repost: Writing a book with Quarto

2025-05-23T06:17:00.003-05:00

Reposted from the original at https://blog.stephenturner.us/p/quarto-books.

...

In the spirit of learning in public, I wanted an excuse to dive into Quarto to learn more about publishing formats beyond simple PDF and HTML documents.

If you’re not familiar, Quarto (quarto.org) is the successor to RMarkdown, the next-generation scientific publishing system that works natively with Python, R, and OJS. If you already have RMarkdown you probably don’t have to do anything to it to get it to render with Quarto. The wonderful thing about Quarto (and to a lesser extent, RMarkdown) is that you can write one single input document and render many types of output documents — HTML, PDF, Word docs, presentations (Powerpoint, Beamer, RevealJS), dashboards, websites, books, blogs, and more). And, Quarto Live, you can embed WebAssembly-powered interactive code blocks for R and Python right into a Quarto document (example here).

I demonstrate here how I turned an old course website of mine made from a bunch of RMarkdown documents into a polished e-book using Quarto. I also briefly point out Quarto Manuscripts and Quarto Dashboards at the end.

You can read the book or download a PDF at https://bdsr.stephenturner.us/.

My old course website

Back when I was faculty at UVA I started a series of workshops in response to the growing demand for practical education in data science and bioinformatics that the traditional coursework at the time lacked. I eventually turned this into a graduate course, and later into a course directed to faculty seeking a career in translational science. The course was a Software Carpentry style live coding hands-on course, mostly using R, that covered topics including data manipulation with dplyr, visualization with ggplot2, predictive modeling with caret, text mining with tidytext, RNA-seq analysis with DESeq2, basic statistics, survival analysis, and other topics.

I made the course website using RMarkdown Websites — a feature that I don’t think ever got much traction, but I found incredibly useful. You put a _site.yml file in the root of your project, and you got a little “Build Website” button in the RStudio build pane. Hit that button and it would render all the RMarkdown documents in the project, and give you a website with pages listed as they are in the _site.yml file. I took a lot of inspiration from Jenny Bryan’s old STAT545 course, and borrowed teaching ideas from other courses and blog posts around the internet.

The old RMarkdown website interface in RStudio.

This actually worked fairly well! My old workshop and course material website is still alive at stephenturner.github.io/workshops (screenshot below), and the code is all open on GitHub (github.com/stephenturner/workshops)

My old course stephenturner.github.io/workshops built as an RMarkdown website.

Quarto books

—Quarto has entered the chat—

Quarto showed up on my radar in 2022 at the last rstudio::conf (before it became posit::conf), where Posit announced the name change, public benefit corp status, Shiny for Python, and, Quarto. I’ve slowly switched most of my technical authoring from RMarkdown to Quarto. I had always been a big fan of the rticles package, and now Quarto is starting to catch up with journal article templates (I wrote the biorecap paper using a generic arXiv Quarto template).

I have bookmarks to so many great reference books including R for Data Science, Hands on Programming with R, and Python for Data Analysis, all of which are written as Quarto books.

The docs (quarto.org/docs/books) looked fairly simple. Just stick a bunch of qmd files in a directory and reference them in a _quarto.yml file. I wanted an excuse to explore the book authoring experience with Quarto, so I grabbed all the source code from my old course website to give it a try.

I intended to write this as a short tutorial, but there’s no tutorial here, because I really didn’t have to do anything! All the RMarkdown source from my old website just worked. There were a few slight customizations I made to the _quarto.yml from my old _site.yml. I updated a little bit of the verbiage to refer to book Chapters instead of lessons, and I updated a few places to use cross-references instead of hard-coded hyperlinks. I followed the GitHub pages publishing docs to configure my gh-pages branch to serve up the content I push to the main branch after running quarto publish gh-pages. And I stuck a custom subdomain on the repo so I could serve at bdsr.stephenturner.us instead of the default stephenturner.github.io/bdsr. The process took me about an hour or so. The book website is live at bdsr.stephenturner.us, and the publish command automatically makes both PDF and EPUB versions available.

My old course website, transformed into a Quarto book, hosted on GitHub Pages. Available at bdsr.stephenturner.us.

Biological Data Science with R

The book is based on course material I developed around 2015 and taught through 2018, so it’s starting show its age. There are some dplyr functions that are deprecated or superseded, it uses gather() and spread() from tidyr instead of pivot_*(), and it’s using caret and related packages instead of tidymodels. And, in the predictive modeling and forecasting chapter, there’s a section on forecasting influenza-like illness that shows perfectly regular seasonal ILI patterns through 2019-2021, a counterfactual that co-instructor and co-author VP (Pete) Nagraj and I published a paper on years later.

Learning more

I recently wrote a short essay about learning in public:

Learning in Public

Stephen Turner

September 24, 2024

Read full story

In that spirit I wanted to share a few resources related to books and other Quarto topics I’ve been reading.

The Quarto Books documentation is a well-organized place to start to get more info on publishing a book with Quarto, and you can find other great examples in the gallery. I also wanted call out two relatively new Quarto output document types that are worth looking at.

Quarto Manuscripts

Mine Cetinkaya-Rundel, Professor of the Practice of Statistical Science at Duke University, gave a great talk about Quarto Manuscripts at the R/Medicine conference earlier this year. The talk was awesome. With Quarto Manuscripts you can write a narrative and include additional R/Python/Etc notebooks alongside the manuscript, and render the output in multiple formats. See the example here, and Mine’s talk below.

Quarto Dashboards

I’ve used and recommended flexdashboard for making static or Shiny-enabled dashboards using RMarkdown. When Quarto first launched, dashboards were missing for at least a year. Quarto 1.4 was released earlier this year, introducing Quarto Dashboards along with other new features. See the short video below from Posit.

Repost: uv, part 3: Python in R with reticulate

2025-05-06T07:48:00.001-05:00

Reposted from the original at https://blog.stephenturner.us/p/uv-part-3-python-in-r-with-reticulate.

Two demos using Python in R via reticulate+uv: (1) Hugging Face transformers for sentiment analysis, (2) pyBigWig to query a BigWig file and visualize with ggplot2.

---

This is part 3 of a series on uv. Other posts in this series:

uv, part 1: running scripts and tools
uv, part 2: building and publishing packages
This post
Coming soon…

Python and R

I get the same question all the time from up and coming data scientists in training: “should I use Python or R?” My answer is always the same: it’s not Python versus R, it’s python and R — use whatever tool is best for the job. Last year I wrote a post with resources for learning Python as an R user.

Python for R users

Stephen Turner

October 21, 2024

Read full story

“The best tool for the job” might require multilingual data science. I’m partial to R for data manipulation, visualization, and bioinformatics, but Python has a far bigger user base, and best to not reinvent the wheel if a well-tested and actively developed Python tool already exists.

Python in R with reticulate and uv

If I’m doing 90% of my analysis in an R environment but I have some Python code that I want to use, reticulate makes it easy to use Python code within R (from a script, in a RMarkdown/Quarto document, or in packages). This helps you avoid switching contexts and exporting data between R and Python.

You can import a Python package, and call a Python function from that package inside your R environment. Here’s a simple demo using the listdir() function in the os package in the Python standard library.

library(reticulate)
os <- import("os")
os$listdir(".")

Posit recently released reticulate 1.41 which simplifies Python installation and package management by using uv on the back end. There’s one simple function: py_require() which allows you to declare Python requirements for your R session. Reticulate creates an ephemeral Python environment using uv. See the function reference for details.

Demo 1: Hugging Face transformers

Here’s a demo. I’ll walk through how to use Hugging Face models from Python directly in R using reticulate, allowing you to bring modern NLP to your tidyverse workflows with minimal hassle. The code I’m using is here as a GitHub gist.

See the code on GitHub

R has great tools for text wrangling and visualization (hello tidytext, stringr, and ggplot2), but imagine we want access to Hugging Face’s transformers library, which provides hundreds of pretrained models, simple pipeline APIs for things like sentiment analysis, named entity recognition, translation, or summarization.1 Let’s try running sentiment analysis with the Hugging Face transformers sentiment analysis pipeline.

First, load the reticulate library and use py_require() to declare that we’ll need PyTorch and the Hugging Face transformers library installed.

library(reticulate)
py_require("torch")
py_require("transformers")

Even after clearing my uv cache this installs in no time on my MacBook Pro.

Installed 23 packages in 411ms

Next, I’ll import the Python transformers library into my R environment, and create a object to use the sentiment analysis pipeline. You’ll get a message about the fact that you didn’t specify a model so it defaults to a DistilBERT model fine tuned on the Stanford Sentiment Treebank corpora.

transformers <- import("transformers")
analyzer <- transformers$pipeline("sentiment-analysis")

We can now use this function from a Python library as if it were an R function.

analyzer("It was the best of times")

The result is a nested list.

[[1]]
[[1]]$label
[1] "POSITIVE"

[[1]]$score
[1] 0.9995624

How about another?

analyzer("It was the worst of times")

Results:

[[1]]
[[1]]$label
[1] "NEGATIVE"

[[1]]$score
[1] 0.9997889

Let’s write an R function that gives us prettier output. This will take text and output a data frame indicating the overall sentiment and the score.

analyze_sentiment <- function(text) {
  result <- analyzer(text)[[1]]
  tibble(label = result$label, score = result$score)
}

Let’s try it out on a longer passage.

analyze_sentiment("it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair")

The results:

     label     score
1 NEGATIVE 0.5121167

Now, let’s create several text snippets:

mytexts <- c("I love using R and Python together!",
             "This is the worst API I've ever worked with.",
             "Results are fine, but the code is a mess",
             "This package manager is super fast.")

And using standard tidyverse tooling, we can create a table showing the sentiment classification and score for each of them:

library(dplyr)
library(tidyr)
tibble(text=mytexts) |>
  mutate(sentiment = lapply(text, analyze_sentiment)) |>
  unnest_wider(sentiment)

The result:

# A tibble: 4 × 3
  text                                         label    score
  <chr>                                        <chr>    <dbl>
1 I love using R and Python together!          POSITIVE 1.00 
2 This is the worst API I've ever worked with. NEGATIVE 1.00 
3 Results are fine, but the code is a mess     NEGATIVE 0.999
4 This package manager is super fast.          POSITIVE 0.995

Demo 2: pyBigWig to query a BigWig file

This example demonstrates using pyBigWig to query a BigWig file in R for downstream visualization with ggplot2. All the code is here as a GitHub Gist.

See the code on GitHub

First, let’s get this example BigWig file:

x <- "http://genome.ucsc.edu/goldenPath/help/examples/bigWigExample.bw"
download.file(x, destfile = "bigWigExample.bw", mode = "wb")

Now let’s load reticulate and use the pyBigWig library:

library(reticulate)
py_require("pyBigWig")
pybw <- import("pyBigWig")

Now let’s open that example file, look at the chromosomes and their lengths, then query values near the end of chromosome 21.

# Open a BigWig file
bw <- pybw$open("bigWigExample.bw")

# Get list of chromosomes
chroms <- bw$chroms()
print(chroms)

# Query values near the end of chromosome 21
chrom <- "chr21"
start <- chroms[[1]]-100000L
end <- start+1000L

# Get values (one per base)
values <- bw$values(chrom, start, end)

# Close the file
bw$close()

Finally, we can put the results into a data frame and plot it with ggplot2:

# Wrap into data frame
df <- data.frame(position = start:(end - 1), 
                 signal = unlist(values))

# Plot the result
library(ggplot2)
ggplot(df, aes(x = position, y = signal)) +
  geom_line(color = "steelblue") +
  theme_minimal() +
  labs(title = paste("Signal at", chrom, "from", start, "to", end),
       x = "Genomic position",
       y = "Signal")

Here’s the resulting plot:

Until recently support for this kind of tooling in R was minimal or non-existent. With new tools like ellmer, mall, and many others now on the scene, R is catching up quickly with the Python ecosystem for developing with LLMs and other AI tools. See my previous post demonstrating some of these tools.

Repost: R 4.5.0 and Bioconductor 3.21

2025-04-17T14:22:00.003-05:00

Reposted from the original at https://blog.stephenturner.us/p/r-450-bioconductor-321.

Faster package installation, import only the functions you want with use(), built-in Palmer penguins data, grep values shortcut, and lots of new bioinformatics packages in Bioconductor

...

R 4.5.0 was released last week, and Bioconductor 3.21 came a few days later. You can read the R release notes here and the Bioconductor 3.21 announcement here. Here I’ll highlight a few things that are worth upgrading for.

R 4.5.0

You can read the R release notes here or just use news().

Faster `install.packages()`

Any time the minor version changes (e.g., 4.4.x to 4.5.x) you’ll need to reinstall all your packages again. Which will now be much faster thanks to under-the-hood changes to install.packages() that downloads packages in parallel:

install.packages() and download.packages() download packages simultaneously using libcurl, significantly reducing download times when installing or downloading multiple packages.

I don’t have any benchmarks to show but I can tell you reinstalling all my go-to packages was much faster than with previous upgrades and reinstalls.

I keep my own “verse” package on GitHub: github.com/stephenturner/Tverse. This “package” is just an empty package with a DESCRIPTION listing all the packages I use most frequently as dependencies (tidyverse, janitor, pkgdown, usethis, breakerofchains, knitr, here, etc). I’ll first install.packages("devtools"), then devtools::install_github("stephenturner/Tverse"), and all the packages I use most frequently are installed because this dummy “verse” package depends on them. You never want to load this library, but it’s an easy way to reinstall all the R package you use frequently on any new machine or with an R upgrade.

Built-in `penguins` data

The built-in iris dataset leaves something to be desired. It’s overused and not very engaging for learners or audiences who’ve seen it repeatedly. It’s small (150 rows), clean (no missing data), and doesn’t present challenges to deal with like outliers and multiple categorical labels. And, if you look at the help for ?iris, you’ll see it was published by Ronald Fisher in the Annals of Eugenics, not something I’d care to cite.

Back in 2020, Allison Horst published the palmerpenguins package — a data package meant to provide a great dataset for data exploration and visualization as an alternative to iris. Previously you had to install the package to use the data, but now the penguins data is built in, as is the less clean penguins_raw data that can be used for teaching data cleaning and manipulation.

library(ggplot2)
ggplot(penguins, aes(bill_len, bill_dep)) + 
  geom_point(aes(col=species, size=body_mass, shape=island)) + 
  theme_classic()

And if you’ve written tutorials using the palmerpenguins package, know that the column names differ between the palmerpenguins and base R versions. You might check out the new basepenguins package (documentation). This package gives you functions to convert all your code over to use the base version.

Import only what you want with `use()`

If you’re coming from Python you’re used to being able to only import what you need from installed packages. E.g.:

from numpy import array, mean
from pathlib import Path

With R if you want to call a function from a package without loading the entire namespace you can always use the double-colon, e.g. dplyr::select(). With R 4.5.0 you can import only the functions you want from any particular package.

use("dplyr", c("filter", "count"))
penguins |>
  filter(island=="Torgersen" & year==2007) |> 
  count(sex)

     sex n
1 female 8
2   male 7
3   <NA> 5

There’s a gotcha here though. As the documentation states, use() is a simple wrapper around library which always uses attach.required=FALSE. That means once you call use() the first time, you need to import all of the functions that you might want to import. If you call use() again or even library(), you still won’t have access to those other functions.

In reality I think I’ll still use the package::function() syntax when I don’t want to load the full library, but it’s nice to have this option. See also the box and conflicted packages.

## This works
use("dplyr", c("filter", "count"))
penguins |>
  filter(island=="Torgersen" & year==2007) |> 
  count(sex)

## This fails!
library(dplyr)
penguins |>
  mutate(ratio=bill_len/bill_dep)

Error in mutate(penguins, ratio = bill_len/bill_dep) : 
  could not find function "mutate"

`grepv()`: shortcut for `grep(..., value=TRUE)`

A very small quality of life improvement. If you use grep(), you get the indices of the vector that match your pattern. To get the actual matched values, you can add value=TRUE, or just use the new grepv() function instead.

x <- c("apple", "banana", "whatsapp")

grep("app", x)
#> [1] 1 3

grepv("app", x)
#> [1] "apple"    "whatsapp"

Bioconductor 3.21

You can read the full Bioconductor 3.21 announcement here. You can install it with:

if (!require("BiocManager", quietly = TRUE))
    install.packages("BiocManager")
BiocManager::install(version = "3.21")

There are 72 new software packages in this release of Bioconductor, bringing the total 2341 software packages, 432 experiment data packages, 928 annotation packages, 30 workflows and 5 books. This release introduces a wide array of new tools and packages across genomics, transcriptomics, proteomics, metabolomics, and spatial omics analysis. Notable additions include new frameworks for spatial transcriptomics (like clustSIGNAL, SEraster, and CARDspa), enhanced utilities for analyzing single-cell data (SplineDV, mist, dandelionR), and cutting-edge methods for integrating and visualizing multi-omics datasets (RFLOMICS, pathMED, MetaboDynamics). The release also features robust statistical and machine learning approaches, such as LimROTS, CPSM, and XAItest, for improved inference and predictive modeling. Several tools improve visualization, accessibility, and reproducibility, including GUI-based apps (geyser, miaDash, shinyDSP) and packages focused on optimizing performance or interoperability (e.g., RbowtieCuda, ReducedExperiment, Rigraphlib).

You may also want to skim through the release notes to look through the updates to existing packages you already use.

Repost: Bluesky conversation analysis with local and frontier LLMs with R/Tidyverse

2024-12-30T09:59:00.002-06:00

Reposted from the original at https://blog.stephenturner.us/p/bluesky-analysis-claude-llama-tidyverse.

Summary, translation, and sentiment analysis of Bluesky posts on a topic using atrrr, tidyverse/ellmer, and mlverse/mall with Claude and other open models, and gistr to put the summary on GitHub.

...

Background

Bluesky, atrrr, local LLMs

I’ve written a few posts lately about Bluesky — first, Bluesky for Science, about Bluesky as a home for Science Twitter expats after the mass eXodus, another on using the atrrr package to expand your Bluesky network. I’ve also spent some time looking at R packages to provide an interface to Ollama. I wrote about this generically last summer, and quickly followed that up with a post about an R package that uses Llama 3.1 for summarizing bioRxiv and medRxiv preprints.

Using frontier models in R

In those posts above I’m using a local LLM (some variant of the llama model). Meta claims that the newest llama3.3-70B model achieves performance on par with GPT-4 and the much larger llama3.1-405B. It’s good (really good for an open-ish model), but I don’t agree with this claim (and neither does the Chatbot Arena leaderboard).

Enter ellmer (ellmer.tidyverse.org) — a recent addition to the tidyverse that provides interfaces to frontier models like OpenAI’s ChatGPT, Anthropic’s Claude, Google’s Gemini, and others (along with local models via Ollama).

I bought a few bucks worth of Claude API credits to see how Claude would compare to some of these open source models for a simple summarization task.

The task

Bluesky for science has really taken off since X’s new management has let the place degrade into the sewer that it is. It’s such a lively place and if I go a day or two without scrolling through my feeds I’ll miss a lot of what’s happening in the R community and other interest groups.

I subscribe to the AI News newsletter, where every day I get an AI-summarized digest of what’s happening in the AI world across Twitter, Reddit, various Discords, and other forums (fora?). I was curious if I could do something similar with the #Rstats hashtag on Bluesky.

Subscribe now

The setup

Here’s the basic setup. Here’s the code on GitHub.

Use the atrrr package to retrieve 1000 Bluesky posts tagged with #Rstats. I typically see about 100 posts per day, so 1000 is enough to capture everything in the past week.
Use dplyr to do some cleaning: Limit posts to just the previous 7 days, create a link to the post URL from the at:// URI, arrange descending by the most liked posts, having at least 5 likes, extract the post text to feed to the LLM, and extract a bullet point list of post text for the top 10 posts to include in the summary.
Use the ellmer package to summarize these posts using Claude 3.5 Sonnet and various open models through Ollama. Run three iterations of each, and choose the best result out of the three (subjective).
1. Claude 3.5 Sonnet
2. Llama3.3
3. Gemma2
4. Mistral
Use ROpenSci’s gistr package to programmatically post the resulting markdown as a GitHub gist.

The system prompt I’m using for each of these is:

What are the top posts on #rstats Bluesky last week? I will provide you with data showing the author, post URL, and post text from the top 100 most liked posts tagged with #Rstats. Remember, this is bluesky, and posts are called posts, NOT tweets. Make sure you call them posts, not tweets. I want you to return a summary of the top posts, in markdown. Who posted about what? And why is it important? Give examples. Give me a list of topics, a list of common hashtags, and author names. Also talk about the general overall tone or sentiment. You can link to a specific post by using the URL provided, where the link text is the author's name, like '[author](url) posted <text>'. Do not include any preample or postscript text or followup question suggestions. Here are the posts.

Here's the code on GitHub

Results

Note: I ran the code linked above to generate these results for the week of December 18-24, 2024. You’ll get different results running the same code today.

The result from Claude 3.5 Sonnet was far and above the best summary, and it followed my system prompt instructions better than any of the open source models. After checking my usage, this query cost about $0.01-$0.02 cents.

Claude 3.5 Sonnet

Results from Claude 3.5 Sonnet:

Summary
Produced with claude-3-5-sonnet-20241022
Here's a summary of the top #rstats posts on Bluesky from the past week:
Key Announcements and News:
Posit (formerly RStudio) joined the Open Source Pledge, committing $2000/year per developer to support open source software
The RStudio IDE and Posit Workbench received a major update with Positron Pro support
useR 2025 will be held at Duke University in August 2024
Several notable package releases including rms 6.9-0, {chopin} for spatial analysis, and {rayrender} v0.38.0
Popular Topics and Themes:
Package Development & Updates
Data Visualization
Tutorials & Learning Resources
Statistical Methods
Mobile Development
Generative Art
Notable Tutorial Content:
Jarrett Byrnes shared a tutorial on Bayesian SEM using brms
Danielle Navarro wrote about regular expressions in R
Multiple posts about the easystats ecosystem for Bayesian analysis
Innovation & New Tools:
Colin announced Rlinguo, a mobile app for running R on phones
New developments in web-based R applications using webR and Shinylive
Various updates to visualization packages and tools
Community Engagement:
Active TidyTuesday participation analyzing D&D spell data
Significant discussion around generative art and visualization
Strong focus on teaching and learning resources
The overall tone of the R community on Bluesky is overwhelmingly positive and collaborative, with a mix of technical discussion, creative applications, and mutual support. There's a strong emphasis on sharing knowledge and resources, particularly around data visualization and statistical methods.
Common Hashtags: #rstats #dataviz #tidytuesday #pydata #statistics #dataBS #rtistry #bayesian #shiny #ggplot2
Notable Authors:
Hadley Wickham
Danielle Navarro
Frank Harrell
Tyler Morgan-Wall
Sharon Machlis
Colin Fay
Nicola Rennie
The community appears to be particularly excited about new developments in mobile and web-based R applications, as well as advances in statistical computing and visualization tools. There's also a strong strain of creativity running through the posts, with many sharing artistic applications of R programming.
Top posts
Top 10 posts:
Hadley Wickham: Very proud to announce that @posit.co has joined the opensourcepledge.com. We're committing to spending $2000 / developer / year to support open source software that we use (but don't develop): posit.co/blog/posit-p... #rstats #pydata
Jarrett Byrnes: OK, here is a very rough draft of a tutorial for #Bayesian #SEM using #brms for #rstats. It needs work, polish, has a lot of questions in it, and I need to add a references section. But, I think a lot of folk will find this useful, so.... jebyrnes.github.io/bayesian_sem... (use issues for comments!)
Danielle Navarro: Some words on regular expressions in #rstats. In which I finally wrap my head around the fact that there are at least three different regex engines in widespread usage in R, they support different features, and they all hate me blog.djnavarro.net/posts/2024-1...
Colin 🤘🌱🏃‍♀️: #RStats I'm so, so thrilled to finally share the release of Rlinguo, a mobile app that runs R 📱 This is a fully, native mobile app that uses #webR in the backend to run R functions. Available on the App Store (iPhone) & Play Store (Android) rtask.thinkr.fr/introducing-...
Erik Reinbergs: These bayes tutorials are the first ones i've actually understood. Looking forward to the rest of the being finished. Go @easystats.bsky.social teams! easystats.github.io/bayestestR/i... #rstats
Dave H: Just pushed a Christmas update to {ggblanket}. I decided to support colour blending. It uses {ggblend} under the hood (thanks @mjskay.com), which uses graphics features developed by Paul Murrell. Give {ggblanket} and {ggblend} a star, if you find them useful. Oh, and have a merry Christmas #rstats
Owen Phillips: I created a thing that puts every player's box score stats from the night before in one interactive table. Ive been using it every morning to quickly check who had a good/bad game instead of looking at 10 different box scores updates nightly #rstats #gt #quarto thef5.quarto.pub/boy/
Hadley Wickham: I discovered that the original plyr website still exists: had.co.nz/plyr/. It's hard to imagine that this used to be a better than average package website #rstats
Frank Harrell: 1/2 Major release 6.9-0 of #RStats rms package now available on CRAN w/complete re-write of the binary/ordinal logistic regression function lrm. lrm began in 1980 as SAS PROC LOGIST so it was time for a re-do for iterative calculations. Details are at fharrell.com/post/mle
Joe Chou: As far as I’m concerned, #webr and #shinylive is indistinguishable from magic. I can’t believe that since first installing #quarto-webr a few hours ago till now, I’ve gotten a relatively complicated #shinyapp exported, deployed, and running serverless. This is magic. #rstats

Open-source models: Llama3.3, Gemma2, Mistral

I’m not showing the other results here — the links go to a Markdown file as a GitHub gist showing results from running the same prompt with Llama3.3, Gemma2, and Mistral. They’re very different! Gemma just provided a very high-level overview that could have come from any collection of posts about R, and mistral just listed out individual summaries of each post instead of summarizing the whole collection. Llama3.3 might have been a bit better, but overall, it seems that the open source models aren’t following my system prompt as well as Claude 3.5 Sonnet is.

Subscribe now

Bonus: sentiment, summary, and translation

While the ellmer package demonstrated above provides a straighforward way to prompt local or frontier models with text, the mall package in the mlverse (mlverse.github.io/mall) provides functions for running LLM predictions against a data frame. It’s API is similar in the R and Python packages.

If I have the data frame created above with the post text in d$text, I can run a sentiment analysis, create a 5 word summary, and translate into Spanish with the code below. New columns are added to the data, prefixed by a dot.

d |>
  llm_sentiment(text) |>
  llm_summarize(text, max_words = 5) |>
  llm_translate(text, "spanish")

Here’s what the top 10 most liked posts from the past week are, summarized in 5 words, translated into Spanish:

Hadley Wickham (posit joins opensourcepledge): Estamos muy orgullosos de anunciar que @posit.co ha unido a opensourcepledge.com. Estamos comprometidos a gastar $2000 / desarrollador / año para apoyar software de código abierto que usamos (pero no desarrollamos): posit.co/blog/posit-p... #rstats #pydata
Jarrett Byrnes (tutorial on bayesian sem rstats): Este es un tutorial muy desgloseado para el uso de SEM bayesiano utilizando brms en RStats. Necesita ser revisado, pulido y tener muchas preguntas resueltas, pero creo que muchos personas lo encontrarán útil, así que... jebyrnes.github.io/bayesian_sem... (usen los problemas para hacer comentarios!)
Danielle Navarro (three r regex engines exist): Hay al menos tres diferentes motores de expresiones regulares en uso amplio en R, que apoyan diferentes características y todos me odian.
Colin 🤘🌱🏃‍♀️ (mobile app for r programming): Estoy extremadamente emocionado de compartir finalmente el lanzamiento de Rlinguo, una aplicación móvil que ejecuta R.
Erik Reinbergs (bayes tutorials finally make sense): Estos tutoriales de Bayes son los primeros que he entendido. Estoy emocionado de ver el resto terminado. ¡Buena suerte a todos en @easystats.bsky.social!
Dave H (pushed christmas update to ggblanket): Acabo de lanzar una actualización de Navidad para {ggblanket}. Decidí apoyar la mezcla de colores. Utiliza {ggblend} detrás del escenario (gracias @mjskay.com), que utiliza características gráficas desarrolladas por Paul Murrell. Dale un estrella a {ggblanket} y {ggblend}, si los encuentras útiles. ¡Que tengas un feliz Navidad #rstats
Owen Phillips (interactive box score table): Crea una cosa que agrega todos los estadísticas de puntuación del tablero de cada jugador de la noche anterior en una tabla interactiva. He estado utilizandola cada mañana para revisar rápidamente quién tuvo un buen/buen juego en lugar de mirar las actualizaciones de diez diferentes partidos nocturnos #rstats #gt #quarto thef5.quarto.pub/boy
Hadley Wickham (original plyr website still exists): Descubrí que la página web original de plyr aún existe: had.co.nz/plyr/. Es difícil imaginar que esto era una página de paquete mejor que promedio. #rstats
Frank Harrell (rms package version released): La versión 6.9-0 del paquete #RStats rms ahora está disponible en CRAN con una completa revisión de la función de regresión logística binomial lrm, que comenzó en 1980 como PROC LOGIST en SAS y requería una reescritura para cálculos iterativos.
Joe Chou (webr and shiny are magic): En mi opinión, #webr y #shinylive son indistinguibles del magia. No puedo creer que desde que instale #quarto-webr unas horas atrás hasta ahora, he logrado exportar, desplegar y correr una aplicación #shinyapp relativamente complicada serverless. Esto es magia. #rstats

Here’s a count of the sentiment classification for each post.

I was curious what the negative sentiment posts were. The most liked negative sentiment post was Danielle Navarro’s blog post on different regex engines in R and the difficulty using them consistently. Others had a degree cynicism that the model picked up on.

"Some words on regular expressions in #rstats. In which I finally wrap my head around the fact that there are at least three different regex engines in widespread usage in R, they support different features, and they all hate me blog.djnavarro.net/posts/2024-1..."
"Found a bug in my code and after fixing it, it stopped working. Wonderful Christmas gift :3 #rstats"
"Turning a 10-minute #rstats project into a two-hour one because I refuse to break my pipe and save an intermediate object to memory."

Subscribe now

Use an LLM to translate help documentation on-the-fly

2024-12-16T07:43:00.004-06:00

Reposted from Paired Ends at https://blog.stephenturner.us/p/llm-translate-documentation.

---

The lang package overrides the ? and help() functions in your R session. The translated help page will appear in the help pane in RStudio or Positron. It can also translate your Roxygen documentation.

---

Using LLMs in R

Most of the developer tooling for AI/LLM training and evaluation is Python-centric, but just over the past few months we’ve seen a surge of new tooling for AI/LLM applications for the R ecosystem.

ollamar and rollama provide wrappers around the Ollama API allowing you to run LLMs locally on your machine. I recently wrote a few posts, one demonstrating how to use ollamar, and another demonstrating a package that uses ollama internally.
Use R to prompt a local LLM with ollamar
Stephen Turner
·
Aug 14
Read full story
biorecap: an R package for summarizing bioRxiv preprints with a local LLM
Stephen Turner
·
Aug 24
Read full story
elmer is a new package in the tidyverse that allows you to interact with many different LLMs via R (Claude, ChatGPT, Gemini, and Ollama too.
mall is an interesting one that provides an easy way to run multiple LLM predictions against a data frame (sentiment analysis, summarization, classification, extraction, translation, etc).
pal provides LLM assistants for doing things like highlighting code and asking for things like roxygen documentation, testthat tests, etc.
Shiny Assistant helps you explain how things work in Shiny, and can help build Shiny apps for you (in either R or Python).

The lang package

The lang package (source, documentation) is an interesting new addition to the mlverse in R. From the documentation:

lang overrides the ? and help() functions in your R session. If you are using RStudio or Positron, the translated help page will appear in the usual help pane.
If you are a package developer, lang helps you translate your documentation, and to include it as part of your package. lang will use the same ? override to display your translated help documents.

Let’s look at an example. I recently invited my colleague and co-author VP Nagraj to write about the rplanes package we published and released on CRAN for plausibility analysis in epidemiological forecasting.

PLANES: Plausibility Analysis of Epidemiological Signals

Stephen Turner

Sep 3

Read full story

One of the first functions you might use from this package is read_forecast(), which reads a probabilistic quantile forecast CSV file for downstream plausibility analysis. Let’s look at the help for this function.

library(rplanes)
?read_forecast

En Español

Now let’s get help in Spanish.1 load the lang package and tell it that we’re using llama3.2.2 We’ll set the system language to Spanish, then ask for help again.

Sys.setenv(LANGUAGE="spanish")
?read_forecast

My fluency in Spanish is limited to general conversation and travel needs so I can’t easily verify the accuracy of the translation of this technical language, but when I ran some of this back through Google Translate it seemed to be mostly faithful. Notice how things that shouldn’t be translated aren’t — function names, arguments, columns in the returned output, code in the examples.

हिंदी में … … باللغة العربية

What about non-Western languages?

Let’s try Hindi!

Sys.setenv(LANGUAGE="hindi")
?read_forecast

I can’t verify the accuracy of this translation beyond running some of the text back through Google Translate, but in doing so at first glance the translation isn’t bad.

What about Arabic?

Sys.setenv(LANGUAGE="arabic")
?read_forecast

If you’re a native speaker of any of these, I’d love to know what you think. Chat with me on Bluesky (@stephenturner.us).

Translating your package’s Roxygen docs

The lang documentation has a great section on using lang as a package developer. You can translate all of your Roxygen documentation into the desired language, then edit those translations by hand as needed. Then a special helper function re-roxygenizes your docs placing them in a special inst/man-lang folder. The lang docs explain how this all works, but once you do this, when a user has the lang package loaded, they’ll get your pre-computed and optionally edited translations instead of having to wait around for the LLM to translate the help.

Demo

Here’s a demo using a very small package I wrote for something completely different. Don’t worry about all the Docker stuff described here. There’s one single function, missyelliot(), that simply reverse complements a DNA sequence (“take that flip it and reverse it”). That is, it’ll convert GATTACA to it’s reverse complement TGTAATC.

Restart your R environment, and install the package using devtools/remotes. Load both rpdd and lang.

devtools::install_github("stephenturner/rpdd")
library(rpdd)
library(lang)

Now get some help for missyelliot(). If your language environment variable is English, you’ll get the English help.

Now, change your system language to Spanish, and try it again. Notice how the translated help is instantaneous — you’re relying on the pre-translated and possibly hand-edited translations that come with the package rather than asking an LLM to translate the help for you on the fly.

Sys.setenv(LANGUAGE = "spanish")
?missyelliot

If your language is set to something without a pre-populated translation, you’ll have to register a model through Ollama and translate in real time.

Sys.setenv(LANGUAGE = "russian")
?missyelliot

I think this might be one of the most impactful applications of LLMs inside a developer environment since the rise and rapid adoption of Copilots. The ability to instantly access documentation in multiple languages through lang represents a significant step forward in making data science more accessible and inclusive for the global R community, breaking down language barriers that have historically made it challenging for non-English speakers to fully engage with R's rich ecosystem of tools and packages.

Turn a GitHub repo into a single text file for LLM-friendly input (repost)

2024-12-09T04:52:00.002-06:00

Reposted from the original at https://blog.stephenturner.us/p/github-repo-to-text-for-llm-input.

---

If you use ChatGPT, Claude, or even some local model through Ollama or HuggingFace Assistants, you’ll know that the chat interface makes it challenging to feed in an entire repo like a Python or R package, because functions, tests, etc. can be scattered across many files throughout a repo. Here I’ll demonstrate how to turn an entire GitHub repo into a single text file for LLM-friendly input, with R and Python packages as examples.

GitHub Repo to Text Converter

This GitHub Repo to Text Converter can help here.

Web app: repo2txt.simplebasedomain.com
Source: github.com/abinthomasonline/repo2txt

This little app takes a URL to a GitHub repo, lets you select which files in the repo directory structure to include, and creates a single plain text file you can feed into an LLM to ask for explanations, usage, or anything else you’d like to ask about the codebase. You can download or copy this text to the clipboard to later paste into an LLM of your choosing.

The tool runs entirely in the browser. You can also provide a personal access token if you want to access private repositories (again, securely, since everything runs in the browser).

Demo: Python package with GPT-4o

I recently demonstrated writing a CLI app using Click from a Cookiecutter template in Python. It’s a simple app that tells you how much caffeine remains in your system based on how much you consume and when you’ll go to bed. It’s on GitHub at stephenturner/caffeinated.

Build a Python CLI with Click+Cookiecutter

Stephen Turner

Nov 10

Read full story

Here I’m pasting the repo URL, https://github.com/stephenturner/caffeinated into the app. The tool is smart enough to recognize that this is a Python package, and I probably want the __init__.py, __main__.py, other package .py files, and the unit tests in tests/ that I’ll use with pytest.

After hitting “Generate Text File” you’ll see that it generates a single plain text file, first listing out the directory structure of the selected files, then listing out the contents of each file separately. Here’s a preview. My __init__.py is empty (just to signal that this is a package), and the cli.py file is truncated here.

Directory Structure:

└── ./
    ├── caffeinated
    │   ├── __init__.py
    │   ├── __main__.py
    │   └── cli.py
    └── tests
        └── test_caffeinated.py


---
File: /caffeinated/__init__.py
---


---
File: /caffeinated/__main__.py
---

from .cli import caffeinated

if __name__ == "__main__":
    caffeinated()


---
File: /caffeinated/cli.py
---

import click
import math
from datetime import datetime
from importlib.metadata import version, PackageNotFoundError

### TRUNCATED ###

Once you have this, you can easily paste this into an LLM of your choice to chat with the codebase. Here, I pasted this into GPT-4o with the leading prompt: “Explain this code to me, how to use it what it does.” GPT-4o returned all of the text and code below as a single response.

First, the directory structure.

GPT-4o explaining the package’s directory structure.

Next, an explanation of __main__.py, cli.py, and the tests.

GPT-4o explaining the key files in this repo.

Next, a demonstration of how to use it. Note above that I did not select the README to add to my text file. GPT-4o divined this usage information from the code itself, rather than regurgitating the README.

GPT-4o explaining example output from this package.

And finally, a simple explanation of the (minimal) test suite.

GPT-4o explaining how to test the package with pytest.

Demo: R package with Claude 3.5 Sonnet

This also works on an R package. In my last job I wrote a package to assist with containerizing R packages with a usethis-like interface. It’s called pracpac (practical R packaging [with Docker]), and you can read more in the paper or on the GitHub repo. Here I’ll take the code from pracpac repo and pull out the relevant R source files.

Generating a single text file from an R package repo.

This time I asked Claude 3.5 Sonnet to tell me more about this package.

Claude 3.5 Sonnet explaining the pracpac R package.

It’s just HTML+JavaScript

If you look at the source code you’ll see that this is a simple implementation using just HTML and JavaScript. Which means you can download the repo and open the index.html, or if you’d prefer a vanity URL, you can fork the repo and set up GitHub pages to serve from your main branch (e.g. stephenturner.github.io/repo2txt).

Tech I'm thankful for (repost)

2024-11-25T09:21:00.000-06:00

Reposted from https://blog.stephenturner.us/p/tech-im-thankful-for-2024

Data science and bioinformatics tech I'm thankful for in 2024: tidyverse, RStudio, Positron, Bluesky, blogs, Quarto, bioRxiv, LLMs for code, Ollama, Seqera Containers, StackOverflow, ...

It’s a short week here in the US. As I reflect on the tools that shape modern bioinformatics and data science it’s striking to see how far we’ve come in the 20 years I’ve been in this field. Today’s ecosystem is rich with tools that make our work faster, better, enjoyable, and increasingly accessible. In this post I share some of the technology I'm particularly grateful for — from established workhorses that have transformed how we code and analyze data, to emerging platforms that are reshaping scientific communication and development workflows.

The tidyverse: R packages for data science. Needs no further introduction.
devtools + usethis + testthat: I use each of these tools at least weekly for R package development.
Rstudio, Positron, and VS Code: Most of the time I’m using a combination of VS Code and RStudio. My first experience with Positron was a positive one, and as several of my dealbreaker functionalities are brought into Positron, I imagine next year it’ll be my primary IDE for all aspects of data science.
Bluesky. This place feels like the “old” science Twitter of the late 00s / early teens. I wrote about Bluesky for Science to get you started. It’s so great to have a place for civil and good-faith discussions of new developments in science, to be able to create my own algorithmic feeds, and to create thermonuclear block/mute lists.
Slack communities. There are many special interest groups and communities with Slack/Discord communities open to anyone. A few that I’m a part of:
- nf-core
- Nextflow
- Bits in Bio
- Data Science Learning Community
- ASeq (Discord)
Blogs. Good old 2000s-era long form blogs. I blogged regularly at Getting Genetics Done for nearly a decade. Over time, Twitter made me a lazy blogger. My posts got shorter, fewer, and further between. I’m pretty sure the same thing happened to many of the blogs I followed back then. In an age where so much content on the internet is GenAI slop I’ve come to really appreciate long-form treatment of complex topics and deep dives into technical content. A few blogs I read regularly:
- Simon Willison’s Weblog: https://simonwillison.net
- One Useful Thing (Ethan Mollick): https://www.oneusefulthing.org
- Ground Truths (Eric Topol): https://erictopol.substack.com
- Asimov Press: https://www.asimov.press
- Century of Biology (Elliot Hershberg): https://centuryofbio.com
- Bits in Bio: https://bitsinbio.substack.com
- Connected Ideas Project (Alexeander Titus) connectedideasproject.com
- Owl Posting (Abhishaike Mahajan): https://www.owlposting.com
- nf-core blog: https://nf-co.re/blog
- R Weekly: https://rweekly.org/
Quarto: The next generation of RMarkdown. I’ve used this to write papers, create reports, to create entire books (blog post coming soon on this one), interactive dashboards, and much more.
Zotero: I’ve been using Zotero for over 15 years, ever since Zotero was only a Firefox browser extension. It’s the only reference manager I’m aware of that integrates with Word, Google Docs, and RStudio for citation management and bibliography generation. The PDF reader on the iPad has everything I want and nothing I don’t — I can highlight and mark up a PDF and have those annotations sync across all my devices. Zotero is free, open-source, and with lots of plugins that extend its functionality, like this one for connecting with Inciteful.
bioRxiv: bioRxiv launched about 10 years ago and every year gains more traction in the life sciences community. And attitudes around preprints today are so much different than they were in 2014 (“but what if I get scooped?”).
LLMs for code: I use a combination of GitHub Copilot, GPT 4o, Claude 3.5 Sonnet, and several local LLMs to aid in my development these days.
Seqera Containers: I’m not a Seqera customer, and I don’t (yet) use Seqera Containers URIs in my production code, but this is an amazing resource that I use routinely for creating Docker images with multiple tools I want. I just search for and add tools, and I get back a Dockerfile and a conda.yml file I can use to build my own image.
Ollama: I use Ollama to interact with local open-source LLMs on my Macbook Pro, for instances where privacy and security is of utmost concern.
StackOverflow: SO used to live in my bookmarks bar in my browser. I estimate my SO usage is down 90% from what it was in 2022. However, none of the LLMs for code would be what they are today without the millions of questions asked and answered on SO over the years. I’m not sure what this means for the future of SO and LLMs that rely on good training data.

Expand your Bluesky network with R (repost)

2024-11-20T03:48:00.000-06:00

This is reposted from the original at https://blog.stephenturner.us/p/expand-your-bluesky-network-with-r.

---

I’m encouraging everyone I know online to join the scientific community on Bluesky.

Bluesky for Science

Stephen Turner

Nov 16

Read full story

In that post I link to several starter packs — lists of accounts posting about a topic that you can follow individually or all at once to start filling out your network.

I started following accounts of people I knew from X and from a few starter packs I came across. One way to expand your network is to take all the accounts you follow, see who they are following but you aren’t. You can rank this list descending by the number of your follows who follow them, and use that list as a way to fill out your network.

Let’s do this with just a few lines of code in R. The atrrr package (CRAN, GitHub, Docs) is one of several packages that wraps the AT protocol behind Bluesky, allowing you to interact with Bluesky through a set of R functions. It’s super easy to use and the docs are great.

The code below does this. It will first authenticate with an app password. It then retrieves all the accounts you follow. Next, it gets who all those accounts follow, and removes the accounts you already follow.1

library(dplyr)
library(atrrr)

# Authenticate first (switch out with your username)
bsky_username <- "youraccount.bsky.social"

# If you already have an app password:
bsky_app_pw <- "change-me-change-me-123"
auth(user=bsky_username, password=bsky_app_pw)

# Or be guided through the process
auth()

# Get the people you follow
f <- get_follows(actor=bsky_username, limit=Inf)

# Get just their handles
fh <- f$actor_handle

# Get who your follows are following
ff <-
  fh |>
  lapply(get_follows, limit=Inf) |>
  setNames(fh)

# Make it a data frame
ffdf <- bind_rows(ff, .id="follow")

# Get counts, removing ppl you already follow
ffcounts <-
  ffdf |>
  count(actor_handle, sort=TRUE) |>
  anti_join(f, by="actor_handle") |>
  filter(actor_handle!="handle.invalid")

# Join back to account info, add URL
ffcounts <-
  ffdf |>
  distinct(actor_handle, actor_name) |>
  inner_join(x=ffcounts, y=_, by="actor_handle") |>
  mutate(url=paste0("https://bsky.app/profile/",
                    actor_handle))

This returns a data frame of all the accounts followed by the people you follow, but who you don’t already follow, descending by the number of accounts you follow who follow them (mouthful right there).

Optional, but you can make this nicer by using the gt package to make a nice table with a clickable link.

# Optional, clean up and create a nice table
library(gt)
library(glue)
top <- 20L
ffcounts |>
  head(top) |>
  rename(Handle=actor_handle, N=n, Name=actor_name) |>
  mutate(Handle=glue("[{Handle}]({url})")) |>
  mutate(Handle=lapply(Handle, gt::md)) |>
  select(-url) |>
  gt() |>
  tab_header(
    title=md(glue("**My top {top} follows' follows**")),
    subtitle="Collected November 19, 2024") |>
  tab_style(
    style="font-weight:bold",
    locations=cells_column_labels()
  ) |>
  cols_align(align="left") |>
  opt_row_striping(row_striping = TRUE)

I can’t embed an HTML file here, but here’s what that output looks like. You can click any one of the names and follow the account if you find it useful.

Maybe you do this iteratively - add your top follows’ follows, then rerun the process a few times to possibly discover unknown second-degree connections.

The code here essentially replicates what @theo.io’s Bluesky Network Analyzer is doing, but all locally using R. That web app is faster and easier to use, and does some smart caching and throttling to avoid API rate limits. See the footnote for more.

Build a Python CLI with Click+Cookiecutter (repost)

2024-11-10T06:06:00.000-06:00

Reposted from the original at https://blog.stephenturner.us/p/python-cli-click-cookiecutter.

---

In the spirit of Learning in Public, I wanted an excuse to explore (1) click for creating command line interfaces, (2) Cookiecutter project templates, and (3) modern tools in the Python packaging ecosystem. If you’re primarily an R developer like me, I recently wrote about resources for getting better at Python for R users.

Python for R users

Stephen Turner

Oct 21

Read full story

Click is a really nice package for creating command line interfaces, and I like it better than argparse or other similar utilities. Simon Willison’s click app cookiecutter template was really helpful in getting the boilerplate set up for a python package, and while I looked at build backends like Flit, Poetry, uv, etc., I ended up just using setuptools with a pyproject.toml.

For this demo I built a silly little Python tool called caffeinated (inspired by coffee-o-clock) that tells you how much caffeine you’ll still have in your system at bedtime based on how much you consume and when. You can install it from PyPI and running caffeinated with the --help option (or without any arguments) prints the help. The code is on GitHub (github.com/stephenturner/caffeinated) if you want to follow along.

Demonstration of installing and running caffeinated (on GitHub and PyPI). First, pip install caffeinated, then run caffeinated --help for usage info. Run caffeinated -c 200 -b 9pm to see how much caffeine will remain in your system if you consume 200mg caffeine right now and go to bed at 9pm.

I’m not totally sure how accurate the formula is here, but I’m using this to calculate how much caffeine remains in circulation (and I’m going with 90mg for “a cup of coffee”).1

N (t) = N_{0} {(\frac{1}{2})}^{\frac{t}{t_{6}}}

Where:

N(t) = Quantity of caffeine remaining
N₀ = Original amount of caffeine
t = Time
t₆ = Coffee's half-life (6 hours)

Click

Click (Command Line Interface Creation Kit) is a Python package for creating command line interfaces in a composable way with as little code as necessary.

Why Click? Why not argparse/docopt/etc? Good questions. The Click documentation has a section on Why Click?, Why not Argparse? and Why not Docopt etc.? I like click because it enables you to easily create command line utilities with subcommands (e.g., mycommand subcommand ... e.g. like bedtools intersect ...), and it supports file handling, makes it easier to handle options versus arguments, and easily supports ANSI coloring of the output.

Simple Click demo

First let’s set up a folder structure we’ll use to create a Python package. Make a new folder named whatever you’re calling the package (in this case, caffeinated), and in that directory create a pyproject.toml file. Create a new subfolder with the same name as the parent directory (caffeinated), and in that folder you’ll have three files. Directory structure should look like this:

caffeinated/
- caffeinated/
  - __init__.py
  - __main__.py
  - cli.py
- pyproject.toml

The pyproject.toml will have just the basics you need for a Python package:

[project]
name = "caffeinated"
version = "0.1.1"
dependencies = ["click"]

[project.scripts]
caffeinated = "caffeinated.cli:caffeinated"

The __init__.py will be empty, and the __main__.py will just have one line that imports the function from the cli.py:

from .cli import caffeinated

if __name__ == "__main__":
    caffeinated()

The cli.py actually has the code for your command line tool. This is a really simple program that just echos out the amount of caffeine you consumed and what time your bedtime is:

import click

@click.command()
@click.option("-c", "--caffeine", default=100)
@click.option("-b", "--bedtime", default=2100)
def caffeinated(caffeine, bedtime):
    click.echo(f"Caffeine consumed: {caffeine} mg")
    click.echo(f"Your bedtime is:   {bedtime}")

Now, pip install the package you just wrote:

pip install .

And now the caffeinated command line utility is ready to use. First, get some help. Notice how by specifying a default value

$ caffeinated --help
Usage: caffeinated [OPTIONS]

Options:
  -c, --caffeine INTEGER
  -b, --bedtime INTEGER
  --help                  Show this message and exit.

Now run it:

$ caffeinated -c 200 -b 2100
Caffeine consumed: 200 mg
Your bedtime is:   2100

The real caffeinated app

You can see the real cookiecutter code here: github.com/stephenturner/caffeinated. Here are links to the actual working code. Everything important is in the cli.py file. It adds a few more arguments, picks up the version from the pyproject.toml, and defines functions to do all the calculation and conveniences such as translating “9pm” into 2100 (hours).

caffeinated/
- caffeinated/
  - __init__.py (empty)
  - __main__.py
  - cli.py
- pyproject.toml

Once you update all the source or just pip install caffeinated again from PyPI, the tool will tell you approximately how much caffeine you’ll have remaining in your system after consuming a certain amount of caffeine at your chosen bedtime. Run caffeinated --help to get help on the options.

Cookiecutter

Organizing your project as a well-structured Python package can streamline development and distribution. The Cookiecutter package provides a straightforward way to generate project templates, ensuring consistency and best practices across your projects. Install it with pip, then I’ll use Simon Willison’s click-app cookiecutter template.

pip install cookiecutter
cookiecutter gh:simonw/click-app

You can see what this looks after running that at this demo. Running this and answering a few of the prompts will create:

The directory structure described above
The __init__.py, __main__.py, and cli.py files with some boilerplate to get started.
A pyproject.toml file based on your answers to the prompts.
A tests directory with boilerplate for writing tests with pytest.
A README with badges pointing to a future PyPI release, changelog from your GitHub releases, license, and test status.
GitHub actions for publishing your tool as a package to PyPI (requires additional configuration as described here).

The final caffeinated code made from Simon’s click-app cookiecutter template is here github.com/stephenturner/caffeinated.

Packaging

setuptools, Flit, Poetry, Hatch, uv

I’ve written scores of R packages for fun and profit. There’s really only one build backend toolchain for R packages that everyone uses: devtools with Roxygen documentation with liberal assistance from usethis.

The Python documentation has a good guide on Packaging Python Projects. The build backend ecosystem in Python is more diverse. I really wanted to take a closer look at Flit, Poetry, Hatch, and others, but because the cookiecutter template that I used created a pyproject.toml using setuptools by default, so I just ran with that.

setuptools is probably the oldest and most widely used packaging tools in Python with good documentation and community support. And with PEP 517, the standard became using a simpler pyproject.toml rather than the old setup.py. You can see my pyproject.toml for caffeinated here. It’s pretty simple, and one key feature is the readme="README.md" entry, which results in the documentation on the PyPI landing page (pypi.org/project/caffeinated/) populated with the README.md in the project root, avoiding the need for duplication.

Flit (flit.pypa.io) looks like a very minimal, very simple build backend for packaging plain Python code. I also took a look at Poetry (python-poetry.org), because if I were building something more complex I think I’d want something to help me manage dependencies instead of having to add them to the pyproject.toml by hand. Poetry helps with this. Finally, there’s a lot of interest in uv (docs.astral.sh/uv) right now. It’s a Python package and project manager written in Rust, and the benchmarks are impressive. See the “uv: Unified Python packaging” blog post for more. uv doesn’t yet have a build backend, but that’s in the works at astral-sh/uv#3957.

Building python packages with setuptools vs Flit vs Poetry vs uv might be the subject of a future post, but for now, I’m just using setuptools+build.

building and deploying with setuptools+build+twine

The pyproject.toml created by the cookiecutter template I used had just about everything I needed to build the package. From here it was simple. This will build the .whl binary file and .tar.gz source packages in a dist/ folder.

python -m build

After this it’s fairly straightforward to upload this to PyPI. But, before uploading to the production pypi.org, you should probably upload to the testing repository (test.pypi.org) first to avoid polluting PyPI with broken or testing packages.

twine upload -r testpypi dist/*

Uploading to the real PyPI follows the same convention. I recommend using a token in your .pypirc file instead of a username/password prompt.

twine upload dist/*

A few seconds later your package will be on PyPI, and you can install it with pip install like you would any other package. The caffeinated package is on PyPI at https://pypi.org/project/caffeinated/.

If you’ve ever tried getting a package into CRAN you know how onerous the process can be and how strict the CRAN maintainers can be2 and you may be shocked to see how easy it is to get a package onto PyPI. There is no curation or review process with PyPI. Upload your source and wheel files with twine and your package is live.

Docker

Once the app is on PyPI it’s easy to create a Docker container. There’s a Dockerfile in the repo that looks like this:

FROM python:3.11-alpine
RUN pip install caffeinated
WORKDIR /files
ENTRYPOINT ["caffeinated"]
CMD ["--help"]

You can build it like this (replace stephenturner with your Docker username):

docker build -t stephenturner/caffeinated

And push it up to Docker Hub:

docker push stephenturner/caffeinated

This container is on Docker hub at stephenturner/caffeinated. Now you can run it:

$ docker run stephenturner/caffeinated -c 200 -b 9pm

You would have 35.4mg of caffeine in your system if you went to bed at 9:00pm (in 15.0 hours).
That's like having 39% of a cup of coffee before bed.

Alternatively, you can easily create an image with your new tool and additional tools you might want in the same container using Seqera containers (see the video linked further below for more details). I created this image that includes both caffeinated and cowsay, and I’m piping the result of caffeinated into cowsay.

$ docker run --rm community.wave.seqera.io/library/pip_caffeinated_python-cowsay:5a33eb2abfe4e6a5 sh -c 'caffeinated --caffeine 200 --bedtime 9pm --start-time 8am | cowsay'
 __________________________________________ 
/ You would have 44.5mg of caffeine in     \
| your system if you went to bed at 9:00pm |
| (in 13.0 hours). That's like having 49%  |
\ of a cup of coffee before bed.           /
 ------------------------------------------ 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Learning more

This is part of my TIL / Learning in Public series, which I wrote about recently:

Learning in Public

Stephen Turner

Sep 24

Read full story

For a quick intro on Click, see the official Click intro video. It’s over 10 years old at this point, it’s mostly still applicable.

This video from NeuralNine demonstrates how to create Click groups to create CLI utilities with subcommands:

This video from ArjanCodes explains how to create a Python Package and publish on PyPI. It uses the old setup.py instead of the more modern pyproject.toml convention, but otherwise it’s still good to understand the steps in the process.

Finally, a little more about easily creating containers with multiple tools for multiple architectures using Seqera Containers:

Python for R users (repost)

2024-10-22T07:03:00.003-05:00

This is reposted from the original at https://blog.stephenturner.us/p/python-for-r-users.

---

A Google search for “R vs Python” returns thousands of hits across sites like Reddit, IBM, Datacamp, Coursera, Kaggle, and many others. A quick Google Trends analysis shows that this search query has grown steadily over the last decade.

Google Trend analysis of “R vs Python” worldwide (gtrendsR+ggplot2).

Any real data scientist would agree that this argument is silly, that the right answer is to use the best tool for the job. What’s “best” isn’t always easy to answer. Sometimes the “best tool” might be the one that you’re most familiar with and can write most easily. After all, compute time is cheap relative to software engineer time. In other cases there truly is a better choice depending on the problem (e.g., the AI ecosystem is generally more well developed in Python, while the Bioconductor ecosystem provides R with a rich framework for many kinds of bioinformatics tasks). I’m partial to R/tidyverse for tabular data analysis and manipulation, but if polars or pandas is your thing, that’s cool too.

Beyond “choose the right tool,” my biggest advice to trainees and students when asked about “R vs Python” was to learn both languages, but learn one of them so well as to be native-level fluent in the language. That is, become so comfortable in a language that you rarely have to look up documentation, where you have a natural intuition about how to best implement a variety of your everyday tasks, where you know how to write code, docs, and unit tests without really thinking about it.

R is that language for me. I can read, write, debug, and package code in Python, but it doesn’t come to me as easily. I’ve been collecting and bookmarking resources over the years that I’ve turned to for writing and reading Python, coming from native R fluency. LLMs have a new place in this as well, and I touch on that in the end.

For beginners: Learn Python with CS50

If you’re just getting started with Python or programming in general, there are thousands of courses and resources to help you get started. My favorite one to recommend is David Malan’s CS50 course at Harvard. The course materials are here, and the entire 16-hour course is available on YouTube. I still revisit the OOP section from time to time. It likely covers concepts you already know well, as this is meant to be an introduction to programming, using Python.

https://www.youtube.com/watch?v=nLRL_NcnK-4

Learn X in Y minutes

learnxinyminutes.com has a collection of very short primers on different programming languages. These assume you’ve programmed in other languages before and understand flow control, OOP, functions, classes, etc. The Learn X in Y minutes where X=Python tour is here: learnxinyminutes.com/docs/python/. It’s a good place to start if you don’t need to learn what a for loop is or how flow control works. This is simply a long Python scripts with comments throughout that demonstrate main features of the language for programmers coming from another language.

Reticulate vignette: Python Primer for R Users

Reticulate is an R package that enables interoperability between R and Python. It allows you to call Python from R, translate between R and python objects, and create bindings to different Python virtual environments or Conda environments.

The Primer on Python for R Users vignette in the reticulate package is a great place to start if you’re a seasoned R user and want to know how to write Python code to do things in R that you’re familiar with. The primer covers data structures, iteration, indexing, function definition, classes, dunder methods, iterators, generators, imports and modules, decorators, and more.

Emily Riederer’s Python Rgonomics series

I’ve never personally met Emily but I’ve been following her since I saw her rstudio::global 2020 talk on building an R package ecosystem internal to your organization (video, slides, post). It’s a great talk whether you’re programming in R, Python, Rust, whatever, and building off of open-source tools packages for internal organization use only.

Emily recently published a three part series on Python ergonomics and patterns for seasoned R developers. Links to her posts with her description below.

Python Rgonomics: Switching languages is about switching mindsets - not just syntax. New developments in python data science toolings, like polars and seaborn’s object interface, can capture the ‘feel’ that converts from R/tidyverse love while opening the door to truly pythonic workflows.
polars’ Rgonomic Patterns: In this follow-up post to Python Rgonomics, we deep dive into some of the advanced data wrangling functionality in python’s polars package to see how it’s powertools like column selectors and nested data structures mirror the best of dplyr and tidyr’s expressive and concise syntax
Base Python Rgonomic Patterns: Getting comfortable in a new language is more than the packages you use. Syntactic sugar in base python increases the efficiency, and aesthetics of python code in ways that R users may enjoy in packages like glue and purrr. This post collects a miscellaneous grab bag of tools for wrangling, formatting (f-strings), repeating (list comprehensions), faking data, and saving objects (pickle).

Additionally, I wasn’t able to make it to posit::conf(2024) this year, but from a few colleagues who did, I heard her talk on the topic was great. The talks from the conference aren’t yet online, but Posit usually posts them to YouTube a few months after the conference. Description of her talk from the conference agenda:

Python Rgonomics
Data science languages are increasingly interoperable with advances like Arrow, Quarto, and Posit Connect. But data scientists are not. Learning the basic syntax of a new language is easy, but relearning the ergonomics that help us be hyperproductive is hard. In this talk, I will explore the influential ergonomics of R’s tidyverse. Next, I will recommend a curated stack that mirrors these ergonomics while also being genuinely truly pythonic. In particular, we will explore packages (polars, seaborn objects, greattables), frameworks (Shiny, Quarto), dev tools (pyenv, ruff, and pdm), and IDEs (VS Code extensions). The audience should leave feeling inspired to try python while benefiting from their current knowledge and expertise.

Python for Data Science translation of R4DS

Hadley, Mine, and Garrett’s book R for Data Science is a must-read for anyone looking to upskill their R chops for data analysis and visualization, and the second edition is available to read free online at r4ds.hadley.nz.

Arthur Turrell and others have written Python for Data Science (aeturrell.github.io/python4DS), heavily inspired by R4DS, with most of the chapters mirroring R4DS. The image below shows the TOCs from both books. The book mostly uses pandas and lets-plot (“ggplot2 for python”).

Tables of contents for Python for Data Science (left) and R for Data Science (right).

Shiny for Python

The Shiny for Python docs (shiny.posit.co/py/docs) are pretty good for getting into Shiny for Python, especially if you have some experience with Shiny for R.

If you’re looking for more of a hand-holding approach, I’d recommend Gordon Shotwell’s free Reactive Web Dashboards with Shiny Course available on Talk Python Training.

Get practice

You don’t get better at Python by reading about Python or watching videos about Python. There’s no lack of good resources for Python exercises. A few resources that I’ve found particularly useful are listed below.

Corey Schafer’s YouTube channel (youtube.com/@coreyms): This channel has playlists of short videos and tutorials on Pandas, Matplotlib, OOP, and many other short and long video walkthroughs I’ve found useful.
Python on Exercism (exercism.org/tracks/python): Lots of free exercises.
Practice Python (practicepython.org): More practice!
Automate the Boring Stuff with Python (automatetheboringstuff.com): A free online book with simple projects for automating boring tasks (renaming files, etc).
Advent of Code (adventofcode.com): An Advent calendar of small programming puzzles for a variety of skill sets and skill levels that can be solved in any programming language you like. It starts every year around the holidays, but you can go back and view old puzzles from previous years any time. These are fun for little friendly office competition as well.

ChatGPT and friends

Of all the advice in this post, this one is likely to get stale fastest, so I’ll keep it short. Use LLMs where you can! ChatGPT and other LLMs can essentially act as your pair programmer or personal StackOverflow helping you translate your R expertise into Python with contextualized explanations. You can ask ChatGPT for Python equivalents of familiar R functions, syntax comparisons, or specific code examples tailored to data manipulation, statistical analysis, or visualization. You can paste in Python code, ask what it’s doing, and see what equivalent R code would have been. Paste in error messages and your code and it’ll tell you what went wrong and how to change things. I imagine all the leading LLMs are better with Python than nearly any other programming language, so make use of these tools where you can!

Getting Genetics Done

Repost: ctrlvee: Extract external R code and insert inline

Repost: OpenAI Codex App: Vibe-updating my old qqman R package to ggplot2 with plan+execute.

The new Codex desktop app

Demo: updating qqman to use ggplot2

Repost: Construct objects with idiomatic R code

Repost: Make your development environment portable and reproducible

Keep your dotfiles in a private GitHub repo

Keep a fresh cloud VM setup script

R: Custom “verse” package on GitHub

Dev containers in VS Code

Using a dev container template

Using a custom Dockerfile

Start VS Code running the container

Using OpenAI Codex in Positron

Positron Assistant: GitHub Copilot and Claude-Powered Agentic Coding in R

Demo: creating an R package with Codex

Why Codex instead of Positron Assistant?

Positron Assistant: GitHub Copilot and Claude-Powered Agentic Coding in R

Positron Assistant

AI code completion in Positron

Code completion with GitHub Copilot

Agent mode to create an R package

R package development in Positron

Repost: Tidy RAG in R with ragnar

GUIs for Local LLMs with RAG

Build a local RAG application with Open WebUI to chat with your Zotero library

Demonstration

Setup

Create a vector store

Retrieval

API cost and local models

Learning more

Repost: The Modern R Stack for Production AI

ellmer: interact with almost any LLM in R

Use R to prompt a local LLM with ollamar

biorecap: an R package for summarizing bioRxiv preprints with a local LLM

Bluesky conversation analysis with local and frontier LLMs with R/Tidyverse

chores: automate repetitive tasks

gander: allow an LLM to talk to your R environment

btw: describe your R environment to an LLM

ragnar: retrieval-augmented generation (RAG) in R

vitals: LLM evaluations in R

kuzco: computer vision in R

mall: use an LLM for NLP on your data

Bluesky conversation analysis with local and frontier LLMs with R/Tidyverse

Other resources

Repost: Writing a book with Quarto

My old course website

Quarto books

Learning more

Learning in Public

Quarto Manuscripts

Quarto Dashboards

Repost: uv, part 3: Python in R with reticulate

Python and R

Python for R users

Python in R with reticulate and uv

Demo 1: Hugging Face transformers

Demo 2: pyBigWig to query a BigWig file

Repost: R 4.5.0 and Bioconductor 3.21

R 4.5.0

Faster install.packages()

Built-in penguins data

Import only what you want with use()

grepv(): shortcut for grep(..., value=TRUE)

Bioconductor 3.21

Repost: Bluesky conversation analysis with local and frontier LLMs with R/Tidyverse

Background

Bluesky, atrrr, local LLMs

Using frontier models in R

The task

The setup

Results

Claude 3.5 Sonnet

Summary

Top posts

Open-source models: Llama3.3, Gemma2, Mistral

Bonus: sentiment, summary, and translation

Use an LLM to translate help documentation on-the-fly

Faster `install.packages()`

Built-in `penguins` data

Import only what you want with `use()`

`grepv()`: shortcut for `grep(..., value=TRUE)`