Insights and Innovations from Posit Conf 2023

Adrian Zetner

2024-01-24

• Sunday, September 17 - Wednesday, September 20, 2023
• Two days of workshops and two days of conference keynotes and talks in “sunny Chicago”
• I’m going to discuss some of the things I learned about that weekend.
• This Posit name is new though, what happened to Rstudio Conf…

• RStudio was founded in 2009 with the vision of creating high-quality open-source software for data scientists with initial focus on the R community
• They invested heavily in open-source development, education, and community growth through their IDE (of the same name), conferences, and package development
• This focus on open-source software often comes into conflict with the imperatives of sustaining a commercial enterprise when shareholder profit trumps customer interests
• In order to avoid those pitfalls RStudio reincorporated as…

• A public benefit corporation with
  • High standards of transparency and accountability
  • Fiduciary responsibility to address social, economic, and environmental needs while still overseeing business goals
• The intent is to remain independent over the long term while remaining committed to open source, broadening focus to be more inclusive of other languages (notably Python), and continue to grow the community
• The RStudio IDE will remain the same
• This conference started out with two one-day workshops on Sunday and Monday before the conference proper began

Workshop 1: Big Data in R with Arrow

• First of the two one-day workshops
• Big Data in R with Arrow
• The workshop focused on using an R interface to Apache Arrow to process larger-than-memory files and multi-file datasets with arrow using familiar dplyr syntax.
• Main reason? …

• To avoid the over use of memory that leads to seg faults and crashing your R session

Big Data in R with Arrow

• What is Arrow?
  • A toolbox
  • Designed to improve
    • Algorithm performance
    • Data transfer efficiency

• A multi-language toolbox for accelerated data interchange and in-memory processing
• It defines a memory format and provides libraries for interaction
• Arrow is designed to both improve the performance of analytical algorithms and the efficiency of moving data from one system or programming language to another

Big Data in R with Arrow

• Accomplished via in-memory columnar format
  • Standardized
  • Language agnostic
• Interaction via Arrow libraries
  • C, C++, C#, Go, Java, JavaScript, Julia, MATLAB, Python, R, Ruby, and Rust.

• Accomplished via an in-memory columnar format for representing structured, table-like data sets in-memory.
• Standardized, language-agnostic specification
• Interaction with this format is via Arrow’s libraries
• Libraries are available for C, C++, C#, Go, Java, JavaScript, Julia, MATLAB, Python, R, Ruby, and Rust.

• The columnar format in question
• Parquet files take advantage of the latest SIMD (Single Instruction, Multiple Data) operations included in modern processors
  • Essentially: Your processor can process more data in parallel across multiple files and portions of files.
• They’re able to do this due to a hybrid contiguous columnar layout used in data storage that improves read speeds over 30x and reduces storage space by >80%
• The only downside is they’re slightly less intuitive to the user

• Here is a comparison to serialization of CSV and pure column based storage formats
• Parquet files use a hybrid method to sequentially store chunks of columns along with limited metadata about them

• This hybrid storage makes them especially useful for both
  • projection: selecting certain columns for the user, does not have to traverse the entire file as it would with CSV
  • predicates: identifying certain rows based on criteria in a specific column, easiest in row-based storage as can be accomplished with sorting
• Data science requires both of these and for efficiency’s sake needs to traverse as little of the file as possible in doing so

• Parquet files leverage metadata to skip parts of the data that can be excluded according to the chosen predicate: for eg. Int < 5
• Skip reading parts of the file or entire files when they are partitioned appropriately
• Parquet files also take advantage of

Parquet

• Further optimizations
  • Run length encoding for duplicate values
  • Dictionary encoding for long strings
  • Projection and predicate pushdown

• RLE encodes sequential duplicates by storing the value and the count of times it consecutively repeats reducing storage and read times
• Like storing factors in R Dictionary encoding replaces values with small integers and stores the mapping separately.
• Select only the necessary columns (projection) and rows (predicate) during the data read process.
• All in all Parquet is a phenomenal data storage format for speed, size, and efficiency
• Why does that matter in the context of this workshop…

Big Data in R with Arrow

• NYC Taxi Dataset
  • BIG data
  • >40Gb on disk
  • > 1.15 billion rows

• This workshop had us examining truly gigantic data
• Too large for loading into memory on just about any machine
• This data was downloaded onto our machines in preparation for the course and was partitioned across multiple folders and files by year and month a process which took hours

Big Data in R with Arrow

• arrow::open_dataset()
  • Pointer to data (in arrow)
  • Build queries as normal with dplyr

• Workhorse is open_dataset
• Call it to point to a directory of parquet files or a single large one and return a Dataset pointer
• Queries are built against that pointer using dplyr verbs as if against a tibble
• Query is not evaluated at this time, nothing moved to memory

Big Data in R with Arrow

• arrow::compute()
• arrow::collect()

• Evaluation of queries is blazingly fast as it relies on C++ libraries
• compute() evaluates the query, in-memory output stays in Arrow
• collect() evaluates the query, in-memory output returns to R

Big Data in R with Arrow

• nrow()
• head(), select(), filter(), and collect()
• across()

• Mostly familiar dplyr verbs for interaction with the results
• nrow() to work out how many rows of data your analyses will return
• compute() when you need to execute intermediate steps
• collect() to pull all of the data into your R session
• head(), select(), filter(), and collect() to preview results
• across() to manipulate data in multiple columns at once

Big Data in R with Arrow

• Efficiently read + filter + join + summarise massive datasets
• Leverage Parquet file format and C++ libraries
• Simple interaction via queries build with dplyr

• Why use Arrow?
• Efficiently read + filter + join + summarise massive datasets
• Leverage Parquet file format and fast C++ libraries
• Simple interaction via queries build with dplyr
• The course included a good amount of practice using Arrow on large datasets
• So far application in our context of mostly SQL databases has been limited

Workshop 2: Shiny in Production

• Second of the two one-day workshops
• Shiny in Production with Golem
• Pretty cool to have Joe Cheng, the original creator of the Shiny framework, as a TA

Shiny in Production

• Shiny is an R package that makes it easy to build interactive web apps straight from R.
• You can host standalone apps on a webpage or embed them in R Markdown documents or build dashboards or any number of things.
• You can also extend your Shiny apps with CSS themes, htmlwidgets, and JavaScript actions.
• Challenges often emerge when these applications are deployed beyond the developer’s machine or developed beyond a minimal size

Shiny in Production

• Problems with performance bottlenecks, crashes, simultaneous user issues, optimization, and un-managed complexity tend to arise when these applications grow beyond their initial ideation
• Part of that is due to many apps being designed as a prototype and turning into a full blown web app
• Even more troubles arise when connecting to external data sources, other execution backends, and incorporating numerous other R packages
• This sort of ballooning complexity makes collaboration, modification, and debugging much more difficult
• All of a sudden your shiny app is looking decidedly less so

Shiny in Production

• Enter the Golem framework for building shiny apps
• Scripts guide you with first steps akin to {usethis} & {devtools}
• Encourages Shiny best practices by providing structure and guardrails
• Produce shiny apps as R packages to ensure dependencies are met

Shiny in Production

• 01_start.R for original setup
• 02_dev.R in for ongoing work
• 03_deploy.R for deployment

├── DESCRIPTION
├── NAMESPACE
├── R
│   ├── app_config.R
│   ├── app_server.R
│   ├── app_ui.R
│   └── run_app.R
├── dev
│   ├── 01_start.R
│   ├── 02_dev.R
│   ├── 03_deploy.R
│   └── run_dev.R
├── inst
│   ├── app
│   │   └── www
│   │       └── favicon.ico
│   └── golem-config.yml
└── man
    └── run_app.Rd

• Dev scripts get you started and keep you rolling.
• They initially create your description, readme and license files, add version control and tests
• Use them to add dependencies, create custom functions, and modularize your shiny code
• When ready the scripts in 03 ease deployment to shiny hosting servers, docker containers, etc through checking code, building the package, and generating appropriate configs

Shiny in Production

• App Scripts live in R/ directory
• Defined front end (UI) and back end (server)

├── DESCRIPTION
├── NAMESPACE
├── R
│   ├── app_config.R
│   ├── app_server.R
│   ├── app_ui.R
│   └── run_app.R
├── dev
│   ├── 01_start.R
│   ├── 02_dev.R
│   ├── 03_deploy.R
│   └── run_dev.R
├── inst
│   ├── app
│   │   └── www
│   │       └── favicon.ico
│   └── golem-config.yml
└── man
    └── run_app.Rd

• UI and Server scripts define the front and back end code of your
• Modules are also added here

Shiny in Production

• App Scripts live in R/ directory

• Modules in individual files
  • Encapsulate and repeat features
  • Prevent collisions
  • Logical organization
  • Easier debugging

├── DESCRIPTION
├── NAMESPACE
├── R
│   ├── app_config.R
│   ├── app_server.R
│   ├── app_ui.R
│   ├── mod_picker.R
│   └── run_app.R
├── dev
│   ├── 01_start.R
│   ├── 02_dev.R
│   ├── 03_deploy.R
│   └── run_dev.R
├── inst
│   ├── app
│   │   └── www
│   │       └── favicon.ico
│   └── golem-config.yml
└── man
    └── run_app.Rd

• Modules allow you to encapsulate distinct app interfaces
• Avoids namespace collisions when using same widget across different areas of your app
• Organize code into logical and easy-to-understand components
• Facilitate collaboration and easy debugging

Shiny in Production

Module UI

picker_ui <- function(id) {

  ns <- NS(id)
  
  tagList(
  
    # UI FEATURES
    
    )
  
}

Module Server

picker_server <- function(id, 
                          sets_rv){
  moduleServer(
    id,
    function(input, 
             output, 
             session) {
      # SERVER LOGIC
    }
  )
}

Easily Added to Apps

# app_ui.R
mod_picker_ui("picker_1")

# app_server.R
input_ids <- mod_picker_server("picker_1")

• An example module for a UI picker element (numbers, groups, etc)
• UI
  • id: String to use for namespace
  • ns <- NS(id): Create proper namespace function
  • HTML taglist of UI elements to return
• Server
  • Logic is encapsulated with namespace applied
• Add to app easily
  • Copy/paste the included module calling functions
  • Use as many times as you like with different names
  • picker_1, picker_2, etc

Shiny in Production

• {profvis}
• {crew}
• Shiny + Arrow
• {shinyloadtest}

• Also covered a number of important aspects including:
  • Profiling your Shiny app with {profvis}
  • Asynchronous processes with {crew}
  • A further exploration of fast data loading/querying with {arrow} & .parquet files
  • Load testing
• Absolutely cram packed day but lots of good content
  • Currently preparing a rewrite of the Surveiller app to take advantage of these improvements

Talks

Abstractions All the Way Down

“An abstraction” is the outcome of a process of deriving general rules and concepts from specific examples — a concept that acts as a common noun for all subordinate concepts and connects any related concepts as a group, field, or category.

In software engineering and computer science, abstraction is the process of generalizing concrete details, such as attributes, away from the study of objects and systems to focus attention on details of greater importance.

• As systems become more complex, we must rely on more abstractions.
• Each abstraction tries to hide complexity, allowing more focus on the more important details
• Mostly don’t care how the results get done layers below the abstraction you’re dealing with, this applies in computer science as well as in organizational structures
  • CS: write code in high level programming language, maybe need to know vaguely how the compiler works in order to debug
  • OS: request from CEO that is passed down corp structure to the people and machines that actually do the work
• Why do we use abstractions? …

Abstractions All the Way Down

Head trunk only hold so much junk

Constrain complexity and separate concerns

All non-trivial abstractions, to some degree, are leaky

• From djikstra: the competent programmer is fully aware of the limited size of his own skull
• With the complexity of the modern world / workplace we cannot be masters of everything
• According to Joe Spolsky who wrote a great article called the Law of Leaky Abstractions: All non…
• Spolsky’s article gives examples of an abstraction that works most of the time, but where a detail of the underlying complexity cannot be ignored, thus leaking complexity out of the abstraction back into the software that uses the abstraction. This results in abstraction failure: unexpected results, errors, incorrect responses

Abstractions All the Way Down: Assertions

• Abstractions can leak so they must be permeable

• No abstraction is right for everyone

• JD’s assertions about abstractions were as follows

Abstractions All the Way Down: Solutions

• Abstraction permeability allows for debugging
  • Communication is key
• “Meet people where they are”
  • 80-16-4 split of normal/advanced/guru users

• … Holds for organizations and computers
  • Communication of owndership, responsibility, code visibilty, etc
• What is fast becoming the ethos of the computation and operational genomics group …
  • Match the abstraction to the person needing
  • don’t jump 0 to 100 and burn down their process to creat an API or a LIMS database before helping troubleshoot and improve the Excel they’re currently having issue with
• Excellent talk, highly recommend

Session Roundup

• Commit to Change: How to Increase Accessibility in Your Favorite Open Source Projects
• ShinyUiEditor: From Alpha to Powerful Shiny App Development Tool
• Parameterized Quarto Reports Improve Understanding of Soil Health
• Magic with WebAssembly and webR
• Automating the Dutch National Flu Surveillance for Pandemic Preparedness

• Call to arms to improve accessibility of docs and packages
• Brilliant tool to allow visual editing of a Shiny UI that creates the server / UI code automatically
• Impressive commitment to customization in reporting based on a few simple rules
• Running shiny without a server in a web browser, frankly mindblowing the possibilities
• Heartening to see the shared challenges we and the Dutch National Flu Surveillance program faced along with similar solutions

Resources

Workshops

• Big Data in R with Arrow
• Intro to Shiny
• Shiny in Production
• Getting Started with Quarto

Talks

• Abstractions all the way down
• Increase Accessibility
• ShinyUIEditor
• Parameterized Quarto Reports
• Magic with WASM and WebR
• Pandem-2
• Full Posit 2023 Playlist