Best Practices in R

Adrian Zetner

2024-01-31

Best Good Enough Practices in R

We’re going to cover a selection of good computing practices that every researcher can adopt, regardless of computational skill. How not to lose your stuff, and generally how to be more efficient. Just as perfect is the enemy of good, striving for best can blind you to improvements you can make today.

Introduction

Seminar Goals

• What to take away from this seminar
  • Ideas, resources, methods to improve future work
• What not to take away from this seminar
  • Any rush to apply these ideas retroactively to all previous projects
• Inspiration and content from sources listed at the end

• What to take away from this seminar
  • Ideas, resources, methods to improve future work
• What not to take away from this seminar
  • Any rush to apply these ideas retroactively to all previous projects
  • Do it while revisiting old projects if you have time and where appropriate

Inspiration and content from sources listed at the end: don’t worry about too many notes

Table of Contents

1. Project Oriented Workflow
2. Readability
3. Reproducibility

Project Oriented Workflow

• A project-oriented workflow is an approach to work that centers around the organization, structure, and management of tasks within the context of a specific, self-contained project.

Why?

• Work on More Than One Thing at a Time

• Team Collaboration

  • Easier concurrent work
  • Easy distribution

• Start and Stop

  • Flexible work schedule
  • Checkpoint to save progress

• Documentation for Continuity

  • Easy resumption
  • Context and guidelines for collaborators

Work on More Than One Thing at a Time

• If waiting for input from a colleague or for an analysis to finish can start another part of the project or another project independently (self-contained).

Collaborate, Communicate, Distribute

• Team Collaboration:
  • Foster collaboration within a team by structuring the project to enable concurrent contributions.
  • Keeping projects self-contained facilitates easy distribution to team members and reporting.

Start and Stop

• Flexible Work Sessions:
  • Embrace a flexible work schedule, allowing for starting and stopping based on availability and focus.
  • Regularly checkpoint the project to save progress.
• Documentation for Continuity:
  • Utilize thorough documentation for easy resumption and provide context and guidelines for others taking over or collaborating.
  • This includes future you!

How?

• Standardized organization of files per project
• Consistent actions

• Dedicated directories with standardized organization of files per project
• Consistent action focused workflow

Organization of Project Directories 📂

Organization of Project Directories 📂

• Project Organization:
  • Folder per project
  • Top-level advertisement
    • RStudio/Git/{here} characteristic files
• Path Construction with here():
  • Utilize here() function
  • here package
  • Paths relative to top-level

• Organize each logical project into a folder on your computer.
• Make sure the top-level folder advertises itself as such. This can be as simple as having an empty file named .here
• Or, if you use RStudio and/or Git, those both leave characteristic files behind that will get the job done.
• Path Construction with here():
• Use the here() function from the here package to build the path when you read or write a file.
  • Create paths relative to the top-level directory

{here} Package 📂

here() displays top-level folder location

library(here)
here()

[1] "C:/Users/azetner/Documents/quarto-presentations"

• here() displays top-level folder location
• Criteria for top level
  • Is a file named .here present?
  • Is this an RStudio Project? Literally, can I find a file named something like foo.Rproj?
  • Is this an R package? Does it have a DESCRIPTION file?
  • Is this a checkout from a version control system? Does it have a directory named .git or .svn?

{here} Package 📂

Build a path to a file in a subdirectory and use it

here("presentations", "20240131-RBP_images", "analysisworkflow.png")

[1] "C:/Users/azetner/Documents/quarto-presentations/presentations/20240131-RBP_images/analysisworkflow.png"

here("presentations/20240131-RBP_images/")

[1] "C:/Users/azetner/Documents/quarto-presentations/presentations/20240131-RBP_images"

arrow.file <- here("presentations/20240124-BWG_images/arrow_dataset.png")

file.info(arrow.file)["size"]

                                                                                                     size
C:/Users/azetner/Documents/quarto-presentations/presentations/20240124-BWG_images/arrow_dataset.png 79441

• Build a path to something in a subdirectory and use it
• This is taken directly from Jenny Bryan’s github ode to the here package
• Now files within the project are easily accessible from a standardized format regardless of where the access began
  • Rmd files look for subdirs under their location which if they’re organized into a docs folder Data won’t be a subdir
• Easily passable to another user / computer
• File locations can be more easily built programmatically
• Continuing on with folder structure…

Folder Structure 📂

• Folder Structure:
  • Data
  • Code
  • Documentation
  • External scripts
  • Outputs
• Start project from root
  • Console or IDE
  • here() for paths

• Separate folders for data, code, documentation, external scripts, and outputs
• Use subdirectories for clarity and organization
• Start R from top-level
  • Use {here} for paths
  • Maintain top-level working directory
    • Absolute paths generated at runtime

• Teams Reactions to show who uses RStudio or not
• RStudio is great for maintaining projects

RStudio Projects 📂

• Maintain project separation

• Settings stored in <NAME>.Rproj.

• Open Project in RStudio:

  • Dedicated R process
  • File browser points to Project directory.
  • Working directory set to Project.

Version: 1.0
RestoreWorkspace: No
SaveWorkspace: No
AlwaysSaveHistory: Default
EnableCodeIndexing: Yes
UseSpacesForTab: Yes
NumSpacesForTab: 2
Encoding: UTF-8
RnwWeave: Sweave
LaTeX: pdfLaTeX
AutoAppendNewline: Yes
StripTrailingWhitespace: Yes
LineEndingConversion: Native
BuildType: Package
PackageUseDevtools: Yes
PackageInstallArgs: --no-multiarch --with-keep.source
PackageRoxygenize: rd,collate,namespace

• RStudio Projects maintain project folder architecture
• RStudio leaves notes to itself in <NAME>.Rproj
  • including settings for starting new sessions, code style choices, and external libraries
• Open Project = dedicated instance of RStudio
  • Dedicated R process
  • File browser pointed at Project directory
  • Working directory set to Project directory
• So you’re working in a project but what does that look like? It consists of a series of…

Consistent Actions 💥

• That define a clear and reproducible workflow for data analysis projects

Everything that matters should be achieved through saved code

• All important objects or figures should be explicitly saved to file, via code
• Save source, not the environment…

Save Source not the Workspace / Environment 💥

• Livestock vs. Pets Analogy from Cloud Computing
  • Livestock: managed in herds, disposable
  • Pets: unique, precious
• Treat R processes like livestock
  • Workspace disposability
  • Non-reproducible workflows lead to heartache
• Explicitly save important objects
• Checkpoints for long generation time objects
• Design away fear of reproducibility

• “Livestock is managed in herds and there is little fuss when individuals are lost or must be sacrificed. A pet, on the other hand, is unique and precious.”
• cultivate a workflow in which you treat R processes (a.k.a. “sessions”) like livestock. Any individual R process and the associated workspace is disposable
• if your workspace is a pet, i.e. it holds precious objects and you aren’t 100% sure you can reproduce them, you are guaranteeing heartache
• Explicitly save important objects generated during analysis
• Checkpoints for long generation time objects:
  • Isolate each computational or time demanding step in its own script and write the precious object to file
  • Can simply reload the object from file while developing downstream scripts
• Design your code to explicitly ensure the environment is reproducible
• What’s the best way to ensure you’re not too precious about your workspaces?

Use a Blank Slate 💥

R --no-save --no-restore-data

• Wipe your workspace every load
• This can be set in RStudio global options
• or via command line
• This ensures you don’t rely on precious objects in your R environment being loaded from the project’s .Rdata files

Restart R often 💥

• Restart R to wipe environment
• Save code not workspaces
• Pressure to reinforce correct behaviours
  • Ensuring source code recreates important artefacts

• Restart R in RStudio with ctrl+shift+F10, Why?

• Restart R to wipe environment

• Save code not workspaces

• Pressure to reinforce correct behaviours

  • Ensuring source code recreates important objects

• Saving code – not workspaces – is incredibly important because it is an absolute requirement for reproducibility. Renouncing .Rdata and restarting R often are not intrinsically important or morally superior behaviours. They are important because they provide constant pressure for you to do the right thing: save the source code needed to create all important artefacts of your analysis.

• Replacing the herd with an identical one is what we’re pushing for. There’s no replacing your precious pets

• What best practices to follow while consistently completing these actions?

Analysis Project Practices ⚡

Use an IDE ⚡

• Use an Integrated Development environment to smooth this workflow, I recommend RStudio
• RStudio, VSCode, VIM, whatever
• Take advantage of code-aware editor that will help you notice simple errors like typos and unclosed brackets as well as offering many was to direct code to a running R process. Using an IDE encourages saving code in source files (R/Rmd) rather than sending commands directly to the console
• Use it to organize your workflow and guide towards best practices
• “Sometimes people resist the advice to use an IDE because it’s hard to incorporate into their current workflow and dismiss it as something “for experts only”. But this gets the direction of causality backwards: long-time and professional coders don’t have good habits because they use an IDE. They use an IDE because it makes it so much easier to follow best practices and build good habits.”
• An IDE only helps you keep organized with how you work on software, but there is much more to it

Software Management ⚡

• Research Code and Software:
  • Varied forms and sizes
  • Includes code processing research data, scripts, and workflows
  • Scriptable languages like R, Python, shell, etc
  • Standalone programs for specific research tasks

There are many different shapes and sizes of research software: - Includes code processing research data, scripts, and workflows: any code that runs in order to process your research data. - Any scriptable language (R, Python, shell, etc.) - Standalone programs for specific research tasks - We have to manage it to have a record of all the steps used to process your data

Software Management ⚡

• What can go wrong with research code?
  • What does code do?
  • Why did we do it this way?
  • No longer works
  • Accuracy at question
• Software projects all can benefit modular code:
  • Readable
  • Reusable
  • Testable

What can go wrong? I don’t remember what this code does I don’t remember why I made this choice This code doesn’t work any more I’m not sure if this calculation is correct

These can all be attenuated by adopting a few Software Engineering practices the main one being writing modular code - Focus on making code that is modular as it makes the code - Readable - Reusable - Testable

Modular code should be built of functions…

Software Management ⚡

• Comment brief explanations

• Functions first

  • Clear inputs and outputs
  • Meaningful names
  • One main task

• Ruthlessly eliminate duplication

  • Functions
  • Data Structures
  • Effort

• A function is a reusable section of software that can be treated as a black box by the rest of the program. This is like the way we combine actions in everyday life. Suppose that it is teatime. You could get a teabag, put the teabag in a mug, boil the kettle, pour the boiling water into the mug, wait 3 minutes for the tea to brew, remove the teabag, and add milk if desired. It is much easier to think of this as a single function, “make a cup of tea”.
• When we’re writing scripts built of functions we should think of it in much the same way and apply certain rules to each
• Give them a brief Description: Short is fine; always include at least one example of how the program is used. Remember, a good example is worth a thousand words: Where possible, the description should also indicate reasonable values for parameters, dependencies, etc
• Build programs out of short, single-purpose functions with:
  • clearly-defined inputs and outputs
  • meaningful names (applies to variables too), considering tab completion and name based on scope (counter can be i, major object should have a name that describes its purpose)
• Functions should have one main task, which can be combined into larger functions or workflows or if they get too complex, broken down again into single task functions
  • “Make a cup of Tea” shouldn’t include “Take out the garbage” but it should contain smaller functions to “Boil Kettle” “prepare cup with teabag” etc
• Eliminate duplication:
• Function Usage:
  • Write and re-use functions.
  • Avoid copy-pasting code.
• Data Structure Utilization:
  • Use data structures like lists where possible over multiple related objects
  • Prefer creating a single structure (e.g., score = (1, 2, 3)) over multiple closely-related variables (e.g., score1, score2, score3).
• Effort:
  • Look for well-maintained libraries before writing your own
  • Seek existing code for specific functions.
  • Explore language-specific library catalogs (e.g., CRAN for R, PyPI for Python).
  • Test libraries before relying on them.

Software Management ⚡

• Split monolithic scripts into smaller, individual, independent portions

• Talk about image

• This software needs something to work on so lets talk about…

Data Management 💽

• Why Data Management?
  • Data loss / corruption
  • Confusion about provenance
  • Version

• Data loss / corruption: anything that makes it unusable
• Confusion about provenance: eg. source? purpose?
• Version: eg. what workflow generated the data?

Data Management 💽

• Data Management
  • Save the raw data
  • Ensure raw data is backed up
  • Create analysis-friendly data
    • Create the data you wish to see in the world
      • Self explanatory naming
      • Open formats
      • Machine readability
    • Export cleaned data that you wish you’d received
  • Record all the steps used to process data

• Save data in its original form to ensure faithful retention for rerunning analyses, recovering from mishaps, and experimenting fearlessly. Consider making raw data read only and resist them temptation to overwrite with cleaned results
• For data that’s impractical to manage this way, document the exact procedure, version details, and other pertinent information when working with large, stable databases.
• Back up raw data in multiple places: if it’s not backed up it doesn’t matter
• Analysis friendly data
  • Replace inscrutable variable and column names with self explanatory, machine-readable alternatives
  • Convert data from closed, proprietary formats to open, non-proprietary formats like CSV for tabular data, JSON, YAML, or XML for non-tabular data
  • Create an ideal dataset by focusing on improving machine and human readability without extensive filtering or adding external information. Prioritize machine readability for easy reuse

Data Management 💽

• Each variable must have its own column
• Each observation must have its own row
• Each value must have its own cell

• Consistency
• Vectorization

• One common type of analysis friendly data is Tidy data
• There are three interrelated rules which make a dataset tidy
  • Each variable must have its own column.
  • Each observation must have its own row.
  • Each value must have its own cell.
• Why bother? There’s a general advantage to picking one consistent way of storing data. If you have a consistent data structure, it’s easier to learn the tools that work with it because they have an underlying uniformity.

There’s a specific advantage to placing variables in columns because it allows R’s vectorised nature to shine. As you learned in mutate and summary functions, most built-in R functions work with vectors of values. That makes transforming tidy data feel particularly natural.

• There are two main reasons to use other data structures:
  • Alternative representations may have substantial performance or space advantages.
  • Specialised fields have evolved their own conventions for storing data that may be quite different to the conventions of tidy data.

Data Management 💽

• In much the same way we don’t want monolithic scripts, we also don’t want data transformation to be an entire black box
• Talk about image
  • Proprietary format to open machine readable
  • Interim scripts put out interim results to be reports for environment recreation. Livestock, not pets

Readability 📑

• Wasn’t sure what to call this section as it covers a number of important topics and frankly this was the best I came up with
• Starting with

Naming Conventions 📑

• File names should be:
  • Machine readable
  • Human readable
  • Optional: Consistent
  • Optional: Play well with default ordering

• File names should be:
  • Machine readable
  • Human readable
  • Optional: Consistent
  • Optional: Play well with default ordering

Machine Readable 📑

• Regex/Globbing Friendly
  • Avoid spaces, punctuation, and accented characters
  • Maintain case sensitivity
• Easy Computation
  • Use intentional delimiters for straightforward computational processes
  • Deliberate delimiter use enhances computational efficiency
    • Dashes for spaces between words
    • Underscores for chunks

20220120_patient-exposure_control.csv
20220120_patient-exposure_treatment.csv
20220215_patient-exposure_control.csv
20220215_patient-exposure_treatment.csv
20220215_patient-info_control.csv
20220310_patient-info_control.csv
20220520_patient-info_treatment.csv
20220805_patient-info_control.csv
20230120_patient-info_treatment.csv
20230215_patient-info_treatment.csv
20230310_patient-exposure_control.csv
20230310_patient-exposure_treatment.csv
20230405_patient-exposure_control.csv
20230405_patient-exposure_treatment.csv
20230405_patient-info_control.csv
20230615_patient-info_treatment.csv
20230710_patient-info_control.csv
20230710_patient-info_treatment.csv
20230805_patient-info_treatment.csv

❯ ls -1 2022*
20220120_patient-exposure_control.csv
20220120_patient-exposure_treatment.csv
20220215_patient-exposure_control.csv
20220215_patient-exposure_treatment.csv
20220215_patient-info_control.csv
20220310_patient-info_control.csv
20220520_patient-info_treatment.csv
20220805_patient-info_control.csv

❯ ls -1 *info*
20220215_patient-info_control.csv
20220310_patient-info_control.csv
20220520_patient-info_treatment.csv
20220805_patient-info_control.csv
20230120_patient-info_treatment.csv
20230215_patient-info_treatment.csv
20230405_patient-info_control.csv
20230615_patient-info_treatment.csv
20230710_patient-info_control.csv
20230710_patient-info_treatment.csv
20230805_patient-info_treatment.csv

• Regular Expression and Globbing Friendly:
  • File names should be crafted to be easily processed using regular expressions and globbing (wildcard) patterns.
  • Consider avoiding spaces, punctuation, accented characters, and maintain case sensitivity for consistency in manipulation.
• Easy to Compute On:
  • Intentionally use delimiters in file names, making them straightforward for computational processes.
  • Deliberate use of delimiters enhances the efficiency of computations.
• Discuss searches shown, impossible with inconsistent file names

Human Readable 📑

• Informative File Names:
  • Include content information in file names
  • Anticipate usage context
• Slug:
  • User-friendly and descriptive filenames
  • 20230710_*patient-info_control*.csv

filedir <- here("presentations/20240131-RBP_images/fakedat/")
flist <- list.files(filedir, pattern = "info")

stringr::str_split_fixed(flist, "[_\\.]", 4)

      [,1]       [,2]           [,3]        [,4] 
 [1,] "20220215" "patient-info" "control"   "csv"
 [2,] "20220310" "patient-info" "control"   "csv"
 [3,] "20220520" "patient-info" "treatment" "csv"
 [4,] "20220805" "patient-info" "control"   "csv"
 [5,] "20230120" "patient-info" "treatment" "csv"
 [6,] "20230215" "patient-info" "treatment" "csv"
 [7,] "20230405" "patient-info" "control"   "csv"
 [8,] "20230615" "patient-info" "treatment" "csv"
 [9,] "20230710" "patient-info" "control"   "csv"
[10,] "20230710" "patient-info" "treatment" "csv"
[11,] "20230805" "patient-info" "treatment" "csv"

• Informative Naming:
  • Ensure that the name of the file contains information about its content.
  • Anticipate the context in which the file will be used, to facilitate understanding
    • If in a context where dates are less important than say, organism, lead with the more appropriate chunk
• Slug Concept:
  • Implement the concept of a slug in your filenames: creating user-friendly and descriptive URLs for better comprehension.
  • A slug is often defined as the part of a URL that identifies a page in human-readable keywords
• Easy parsing due to consistent use of delimiters

Easy Sorting 📑

• Numeric Inclusion:
  • Often for code
  • Include a numeric element for effective sorting
  • Left-pad with zeros for consistent width and visual sorting.
  • eg 01_import.R
• Dates:
  • Utilize the ISO 8601 standard for date formatting: YYYYMMDD
  • Ensures chronological sorting in file names by default
  • eg 20220820_wedding-photos.zip

• Numeric Inclusion:
  • Include something numeric in the file name for effective sorting of scripts.
  • Left-pad numeric elements with zeros to maintain a constant width for visually pleasing and consistent sorting.
• ISO 8601 Standard for Dates:
  • Use the ISO 8601 standard for dates in file names to ensure chronological sorting by default.
• Common-Sense Ordering:
  • Consider common-sense ordering based on the nature of the data or content.
  • Ensure that the file names sort logically for easier retrieval.

Naming Conventions 📑

• Avoid:
  • Internal sequential numbers: result1.csv, result2.csv
  • Manuscript locations: fig_3_a.png

• Avoid:
  • Sequential numbers: result1.csv, result2.csv
    • Difficult to parse, non-standard
  • Manuscript locations: fig_3_a.png
    • Liable to change

Writing Meaningful Comments 💬

Programs must be written for people to read, and only incidentally for machines to execute.

Meaningful Comments 💬

Rule 1: Comments should not duplicate the code

if (x > 3) {
   …
} # close if

i = i + 1 # Add one to i

• Stack Overflow blog in resources with 10 rules, lets consider 3
  • Rule 1: Comments should not duplicate the code.
    • Training wheels carried over from early learning exercises
    • add visual clutter
    • waste time writing, reading, and updating for no gain
    • It adds no information whatsoever and incurs a maintenance cost.

Meaningful Comments 💬

Rule 2: Good comments do not excuse unclear code.

getBestChildNode <- function(node) {
  n <- NULL  # best child node candidate
  for (child_node in node$children) {
    # update n if the current state is better
    if (is.null(n) || utility(child_node) > utility(n)) {
      n <- child_node
    }
  }
  return(n)
}

getBestChildNode <- function(node) {
  bestNode <- NULL
  for (currentNode in node$children) {
    if (is.null(bestNode) || utility(currentNode) > utility(bestNode)) {
      bestNode <- currentNode
    }
  }
  return(bestNode)
}

- Rule 2: Good comments do not excuse unclear code.
    - For example generic variable names (x, y, etc) explained in comments
    - Need for comments is reduced if variables are named properly

Meaningful Comments 💬

Rule 3: If you can’t write a clear comment, there may be a problem with the code.

• Rule 3: If you can’t write a clear comment, there may be a problem with the code.

“Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”

• Brian Kernighan
  • If you can’t explain it simply, you don’t understand it well enough
  • Rewrite the code to something you understand well enough to explain or, better yet, that is straightforward.

Roxygen2 Comments 💬

Roxygen2 allows you to avoid the awful practice of documenting packages where you write .Rd files by hand.

Roxygen2 Comments 💬

• In-line documentation
• Rich, dynamic .Rd file generation
• Standard markdown

#' Add together two numbers
#' 
#' @param x A number.
#' @param y A number.
#' @returns A numeric vector.
#' @examples
#' add(1, 1)
#' add(10, 1)
add <- function(x, y) {
  x + y
}

% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/add.R
\name{add}
\alias{add}
\title{Add together two numbers}
\usage{
add(x, y)
}
\arguments{
\item{x}{A number.}

\item{y}{A number.}
}
\value{
A numeric vector.
}
\description{
Add together two numbers
}
\examples{
add(1, 1)
add(10, 1)
}

There are a few advantages to using roxygen2 : - Code and documentation are co-located. When you modify your code, it’s easy to remember to also update your documentation. - roxygen2 provides a number of tools for sharing content across documentation topics and even between topics and vignettes. - There’s a lot of .Rd boilerplate that’s automated away. - You can use markdown, rather than having to learn a one-off markup language that only applies to .Rd files. In addition to formatting, the automatic hyperlinking functionality makes it much, much easier to create richly linked documentation.

Roxygen2 Comments 💬

• Allow for easy bundling of inline documentation into a website

Reproducibility 🐣

Modular and Reusable Code 🐣

• Start programs/functions with explanatory comments.
• Use meaningful names for functions and variables.
• Explicitly state dependencies and requirements.
• Avoid commenting/uncommenting code sections for program control.
• Include a simple example or test data set for clarity.

• As we’ve already discussed
• Writing functions and scripts for modularity
  • Place a brief explanatory comment at the start of every program or function
  • Give functions and variables meaningful names.
  • Make dependencies and requirements explicit.
  • Do not comment and uncomment sections of code to control a program’s behavior.
    • If it’s going to be shared
    • Use proper conditionals or remove
  • Provide a simple example or test data set.
• What is the ultimate version of this process?

Make an R Package 🐣

• Why make an R package?
  • Code Organization
  • Consistent documentation
  • Code Distribution
  • Dependency handling

• Creating R packages for code reuse and distribution Why make an R package?

1. Code Organization: I am always trying to figure out where that “function” I wrote months, weeks, or even days ago. Often times, I end up just re-writing it because it is faster than searching all my .R files. An R package would help in organizing where my functions go.
2. Consistent documentation: I can barely remember what half of my functions do let alone the inputs and outputs. An R package provides a great consistent documentation structure and actually encourages you to document your functions.
3. Code Distribution: No more emailing .R scripts! An R package gives an easy way to distribute your code for others. Especially if you put it on GitHub.
4. Dependency handling: No more failing scripts because library calls failed on load. Install the package and all of its dependencies are installed alongside it!

• Even if it’s just for yourself

Make an R Package 🐣

• Package directory structure
  • Scripts in R
  • Documentation in man
  • README.md
  • DESCRIPTION
  • NAMESPACE
• {Roxygen2} for documentation
• {devtools} for process

.
├── DESCRIPTION
├── NAMESPACE
├── R
│   ├── cat_function.R
│   ├── cat_images.R
│   └── cats-package.R
├── README.md
└── man
    ├── add_cat.Rd
    ├── cat_function.Rd
    ├── cats.Rd
    ├── get_cat.Rd
    └── here_kitty.Rd

• R packages are easy
• Making an R package involves creating a specific structure of directory and adding specific files
• Package directory structure
  • Scripts in R
  • Documentation in man (automated)
  • README.md
  • DESCRIPTION
  • NAMESPACE (automated)
• Easily accomplished with Roxygen2 and devtools
• This is the directory structure of the cats package linked in resources

Make an R Package 🐣

• GH for the cats package from HP’s very approachable “Writing an R package from scratch” blog entry
• Readme rendered from markdown

Make an R Package 🐣

• DESCRIPTION file
  • Dependencies
  • Package metadata
  • Contact information

Package: cats
Title: Cats
Version: 0.1
Author: Hilary Parker <hilary@etsy.com> [aut, cre]
Maintainer: Hilary Parker <hilary@etsy.com>
Authors@R: c( person("Hilary", "Parker", email = "hilary@etsy.com", role =
    c("aut", "cre")))
Description: Mew.
Depends:
    R (>= 3.0.2)
Imports:
    httr,
    ggplot2,
    jpeg
License: MIT
LazyData: true
Suggests:
    testthat

• The job of the DESCRIPTION file is to store important metadata about your package.
• Early on, use these metadata to record what packages are needed to run your package.
• and information about it
• Later, if released publicly let people get in touch with you!

Make an R Package 🐣

• NAMESPACE file
  • Functions from your package
  • Imported functions from others

# Generated by roxygen2 (4.0.1.99): do not edit by hand

export(add_cat)
export(cat_function)
export(get_cat)
export(here_kitty)
import(ggplot2)
import(httr)
import(jpeg)

• The NAMESPACE file specifies which functions your package makes available for others to use and, optionally, imports functions from other packages.

• alt-tab to terminal to show package contents?

• Already showed a quick picture of github which is a bit of a teaser for our next section…

Version Control 🧭

Why version control in data analysis projects?

• File Versioning
• Figure Recreation
• Code Modification Impact
• Directory Copying Fear
• Shared File Row Duplication
• Lost Data Files
• Analysis Choice

• Manuscript Collaboration Merge
• Accidental Deletion
• Project Status Recall
• Experiment Mistake Identification
• Directory Pollution
• Selective Changes
• Ad Infinitum

I have fifteen versions of this file and I don’t know which is which I can’t remake this figure from last year I modified my code and something apparently unrelated does not work anymore I have several copies of the same directory because I’m worried about breaking something Somebody duplicated a record in a shared file with samples You remember seeing a data file but cannot find it anymore: is it deleted ? Moved away ? I tried multiple analysis and I don’t remember which one I chose to generate my output data I have to merge changes to a paper from mails with collaborators I accidentally deleted a part of my work I came to an old project and forgot where I left it I have trouble to find the source of a mistake in an experiment My directory is polluted with a lot of unused/temporary/old folders because I’m afraid of losing something important I made a lot of changes to my paper but only want to bring back one of paragraph

Many Reasons

Version Control 🧭

Version Control 🧭

• What to Save:
  • Human-created content.
  • Plain text files.
  • Avoid binary files.

• What to save?
  • Back up (almost) everything created by a human being as soon as it is created
  • Optimized for plain text files (scripts) and can pinpoint changes between versions
  • Doesn’t work well for binary files like PDFs or MS Office files
  • Raw data shouldn’t change and intermediate files should be able to be reproduced by saved source so no need to change

Version Control: Manual 🧭

CHANGELOG.txt

## 2016-04-08

* Switched to cubic interpolation as default.
* Moved question about family's TB history to end of questionnaire.

## 2016-04-06

* Added option for cubic interpolation.
* Removed question about staph exposure (can be inferred from blood test results).

Add a file called CHANGELOG.txt to the project’s docs subfolder and make dated notes about changes to the project in this file in reverse chronological order (i.e., most recent first). This file is the equivalent of a lab notebook, and should contain entries like those shown.

Version Control: Manual 🧭

Backup entire project folder

.
|-- project_name
|   -- current
|       -- ...project content as described earlier...
|   -- 2016-03-01
|       -- ...content of 'current' on Mar 1, 2016
|   -- 2016-02-19
|       -- ...content of 'current' on Feb 19, 2016

… please don’t do this

Copy the entire project whenever a significant change has been made (i.e., one that materially affects the results), and store that copy in a sub-folder whose name reflects the date in the area that’s being synchronized. This approach results in projects being organized as shown above

Version Control: VCS 🧭

• Version Control System
• Efficient Backup Storage
• Automated Timestamps
• Automated Changelog and Accuracy
• Conflict Resolution and Merging
• Examples: Git, SVN, Bitbucket, etc

• VCS
• Much better than manual
• Efficient Backup Storage:
  • Version control eliminates the need for users to create backup copies of the entire project.
  • Safely stores sufficient information to recreate old versions of files on demand.
• Automated Timestamps:
  • Timestamps all saved changes automatically, eliminating the reliance on users to choose sensible names for backup copies.
• Automated Changelog and Accuracy:
  • Version control systems prompt users whenever a change is saved, ensuring a disciplined approach without relying on manual changelogs.
  • Maintains a 100% accurate record of actual changes made, valuable for troubleshooting later.
• Conflict Resolution and Merging:
  • Instead of blindly copying files to remote storage, version control checks for potential overwrites of others’ work.
  • Facilitates conflict identification and merging changes, ensuring collaboration without loss of data.
• Links to very good tutorials in the resources section of this talk

Version Control: VCS 🧭

Two prominent VCS websites that both use Git

Version Control: VCS 🧭

A side benefit of using these VCS systems and sites is easy hosting of content, from package documentation to presentations like this one

• Now that we’re tracking our code how do we make sure it continues to work?

Managing Project Dependencies 📦

The only way to be certain that code written by someone else will run on your machine and will produce the same results is to replicate their environment

managing project dependencies is a significant issue

Managing Project Dependencies 📦

• What is the environment?
  1. The version of the software used
     - R 3.6.1 vs R 4.3.2
  2. The versions of the packages/extensions used
     - {dplyr} 1.0.10 vs 1.1.12
  3. External software dependencies for packages
     - pandoc

• What is the environment?
  1. The version of the software used
  2. The versions of the packages/extensions used
  3. External software dependencies for packages (eg. pandoc)
• Challenges associated with package versions and dependencies in R projects
  • Often your code will run without any hiccups even when environments differ. Software engineers try to make sure your code will not stop working if you use a slightly different versions.
  • But with all the software and all the dependencies eventually something will give and things will break. Even if the code runs successfully, there is a chance that the results could differ.
• Manually comparing environments is difficult to impossible, using containers to replicate another’s computer is complex so we recommend a mid-point: ensuring package and R versions are the same each time the analysis is run

Resources

http://tinyurl.com/COGseminar

Seminar Content

• What they forgot to teach you in R
• Version Control with Git and Github
• Good Enough Practices in Scientific Computing - Paper
• Good Enough Practices in Scientific Computing - Course
• The Turing Way
• Project Oriented Workflow Slides
• My Previous R Course
• R for Data Science

Moodle Cources

• Introduction to Git
• Other R Courses

Readability

• Code Tells You How, Comments Tell You Why
• Best practices for writing code comments
• Roxygen for in-line documentation
• Quarto for reports
• pkgdown to compile into websites

Packages

• R Packages
• Your first R package in 1 hour
• Writing an R package from scratch
• Making Your First R Package

Project Dependencies

• renv Documentation
• Reproducible R Toolbox
• Reproducibility with renv

Useful Links

• Cheatsheets
• Hacker Laws
• Reproducible Reporting with R