R Package Quality & Validation:
Current Landscape
Key Info
Posit Validation Info
Create R version & Package plan:
Define process of OS implementation within an organization such as R versions & process for adding packages
Pharmas
- Roche
- “Clinical trial reporting is at a pivotal moment today transitioning away from propriety languages. At Roche this has meant freezing our legacy macros library, and we can now celebrate having R as the default backbone for evidence generation in late-stage (with space for more OS languages!).” James Black
- OS NDAs
Do we have to Validate?
A popular question is if regulatory bodies require “validated” packages. The R FDA Submission Pilot used as-is CRAN packages as outlined below. This submission was accepted successfully by the FDA. https://github.com/RConsortium/submissions-pilot1/blob/main/renv.lock https://github.com/RConsortium/submissions-pilot1
Part 1 Posit Team
Part 1
Validation Info
Part 2
R
Part 2
A Guidance Document for the Use of R in Regulated Clinical Trial Environments
Part 3
- A. Public R Packages
- B. Internal Packages (User Functions) https://adv-r.hadley.nz/functions.html
Part 3 A (Public Packages)
There are 2 key things to focus one when it comes to packages in a clinical environment:
- Stable repositories inside the organization
- Users documenting the packages used for clinical reporting
Posit Packages
RStudio in 2020, in collaboration with the community and pharma, released validation guidance documents covering tidyverse, tidymodels, r-lib, gt, shiny and rmarkdown. The main page and direct links are below:
Open Source Package Development & Maintenance
For R packages, maintenance is open and in public view. This can be very helpful for many Quality Teams as it makes an abundance of information available that is often hidden for commercial software. Below we will review this for the ggplot2 package available here: https://github.com/tidyverse/ggplot2
- It is important for Quality Professionals to have a high-level understanding of version control (git etc.) and the maintenance of public open source code via tools like github.
CRAN Checks
Getting a package added to CRAN is no small feat (pass R CMD CHECK & survive the review process). CRAN maintainers test packages across a matrix of R versions & operating systems: https://cran.r-project.org/web/packages/policies.html & https://cran.r-project.org/web/packages/submission_checklist.html
Any package accepted on CRAN must pass a series of automated tests: https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Checking-packages
Here are the results for ggplot2
ggplot2
Code Example
Author Tests
Additional Tests
Many package authors have added their own tests which help ensure that the package’s functions work as intended. Most often, these tests are built into the package in a specific way so that whenever the automatic CRAN checks are run the custom tests also run. These additional tests are part of the package source and for more information please review: http://r-pkgs.had.co.nz/tests.html#test-cran & https://testthat.r-lib.org/
Validation Hub’s Risk-Based Approach guidance
In addition to the Posit Packages (ggplot2 https://posit.co/products/open-source/rpackages/ etc.), pharmaceutical organizations will also add other packages to the baseline of packages to be used by statistical programmers. Organizations will often select other packages to be included based on the R Validation Hub’s Risk-Based Approach guidance here: https://www.pharmar.org/white-paper/
riskassessment demo
Regulatory R Package Repository
Community Work is underway by the R Validation Hub to help bring availability of validated packages for clinical trials. This includes support for the test and validation of public packages commonly used. You can follow this effort here: https://pharmar.github.io/regulatory-r-repo-wg/
Pharmaverse
- https://pharmaverse.org/
- The Pharmaverse is built and maintained by many professional statistical programmers and developers at Roche, GSK, JnJ, Merck and other pharmas. You can read more about it here:https://posit.co/blog/pharmaverse-packages-for-clinical-reporting-workflows/
- The Pharmaverse packages are publicly maintained and the test-coverage and code is available such as: https://github.com/pharmaverse/admiral/tree/main/tests/testthat
Internal Repository of Packages for Clinical Trials
The documents above, and the implemented risk-based approach, allow organizations to establish a baseline of existing packages that are deemed validated and controlled for internal use. Organizations can use Posit Package Manager mentioned above to create an internal repository of packages selected specifically by Quality and IT for clinical trials. Steps to do this are outlined here: https://github.com/philbowsher/Creating-an-Internal-Repository-of-Validated-R-Packages
renv for Statistical Programmers
To help document the packages used and for collaboration with colleagues, statistical programers will use a R package called renv: https://rstudio.github.io/renv/articles/renv.html renv is a companion package used by users in conjunction with the IT best practices used for managing packages. renv creates a file called renv.lock that describes the state of the users project’s library at some point in time and is created with renv::init(). renv.lock records repositories, package versions and can be versioned with git! Here is the R FDA Pilot Submission renv.lock!: https://github.com/RConsortium/submissions-pilot1/blob/main/renv.lock
Part B (Internal Packages)
Testing R Functions:
- thevalidatoR - https://github.com/insightsengineering/thevalidatoR
- valtools - https://github.com/phuse-org/valtools
- testthat - https://testthat.r-lib.org/
Test and Reproducibility for Shiny
- shinytest - https://rstudio.github.io/shinytest/
- {shinyValidator} - http://opensource.nibr.com/shinyValidator/
- rhino - https://appsilon.github.io/rhino/
- golem - https://engineering-shiny.org/golem.html
- shinymeta - https://github.com/rstudio/shinymeta
- TEAL - http://examples.pharmaverse.org/app/teal/
Alternative Approach - Buying Validated Packages
Some organizations prefer to buy validation documentation for existing open source packages. There are various vendors selling validation tests and documents as well as groups selling services to support validation. One such group is Atorus that provides OpenVal which is a subscription-based repository of nearly 200 validated R packages and growing.
Thank you!
PharmaSug Conf
R Package Quality & Validation: Current Landscape