Notation

This document is intended to be extensive and is targeted for both lesson maintainers and the infrastructure team. Because there are overly-detailed descriptions of the infrastructure that maintainers are not required to know, I will label these points with [i] so that maintainers can skip these sections if needed.

Accessibility

With a few exceptions, our lessons have shown to be largely accessible thanks to the reliance on modern frameworks for hosting our lessons. That being said, there are the occasional issues and the WCAG 2.1 accessibility guidelines provides a roadmap to ensure that our lessons are Perceivable, Operable, Understandable, and Robust.

Lesson Folder Organization

The lesson template will be organized such that it clearly separates the lesson content from the lesson style:

|-- .gitignore               # - Ignore everything in the site/ folder
|-- .github/                 # - Scripts used for continuous integration
|   `-- workflows/           #
|       |-- deploy-site.yaml # -   Build the source files on github pages
|       |-- build-md.yaml    # -   Build the markdown files on github pages
|       `-- cron.yaml        # -   reset package cache and test
|-- episodes/                # - PUT YOUR MARKDOWN FILES IN THIS FOLDER
|   |-- data/                # -   Data for your lesson goes here
|   |-- figures/             # -   All static figures and diagrams are here
|   |-- files/               # -   Additional files (e.g. handouts) 
|   `-- 00-introducition.Rmd # -   Lessons start with a two-digit number
|-- instructors/             # - Information for Instructors
|-- learners/                # - Information for Learners
|   `-- Setup.md             # -   setup instructions (REQUIRED)
|-- profiles/                # - Learner and/or Instructor Profiles
|-- site/                    # - This folder is where the rendered markdown files and static site will live
|   `-- README.md            # -   placeholder
|-- config.yaml              # - Use this to configure commonly used variables
|-- CONTRIBUTING.md          # - Carpentries Rules for Contributions (REQUIRED)
|-- CODE_OF_CONDUCT.md       # - Carpentries Code of Conduct (REQUIRED)
|-- LICENSE.md               # - Carpentries Licenses (REQUIRED)
`-- README.md                # - Introduces folks how to use this lesson and where they can find more information.

Episode Structure

All episodes for the Carpentries should be stand-alone markdown documents that can render to valid HTML via pandoc without external dependencies. The structure of an episode is largely free-form, but there are certain elements that should be included to ensure a valid episode in the form of yaml content, required information blocks, instructor notes, and properly closed fenced div tags.

As a refresher, a fenced div tag is a line that starts with at least three colons followed by the name of the tag. The tag is closed by adding another line that starts with at least three colons with no tag at the end:

::::::::::::::::::::::::::: challenge

This is a *challenge block*

::::::::::::::::::::::::::::::::::::::

title: "Creating Examples: Yes, These *Are* Difficult"
teaching: 20
exercises: 10

Error:
    File: '/path/to/episode.Rmd'
    
    The Questions block should be at the top of the document (lines AA--BB).
    Instead, they are located at lines XX--YY. Please edit '/path/to/episode.Rmd'
    and move the block at lines XX--YY to the top of the document (lines AA--BB).

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: instructor

## SLOW DOWN

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

# Lesson 1


This is an example lesson.

:::::::::::::::::::::::::::::::::::::: objectives

 - Write good 

:::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::: questions

 - Is it okay if the number of colons don't match up?
 - Are the sections visible?

::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: instructor

## Example

This is an example of how you write an instructor tag

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


Let's try a challenge

:::::::::::::::::::::::::::::::::::::: challenge

Write a solution

::::::::::::::: solution

NaCl + H2O

:::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::

Basic Infrastructure for Maintainers

To manage and render lessons, you need to have the following software installed. All dependencies for the R packages should automatically install.

• Required
  • R (>= 3.0.1) and associated packages:
    • {renv}
    • {sandpaper}
    • {varnish}
  • RStudio (>= 1.4) OR pandoc (>= 2.11)
  • A Modern Web Browser
• Recommended
  • Git (>= 2.0)
  • Anconda (for Python-based lessons)

The Source of Truth: Dependency Management

Both the Python and the R package ecosystem are constantly evolving, which can often create different outputs from one maintainer’s computer to the other. Moreover, if you maintain a lesson, you may not want to update a package that you are using for your thesis work. To alleviate these problems, we will use the {renv} R package for managing dependencies in the lessons.

When you first start R to preview the lessons, if {renv} is not installed, it will ask you if you want to install it (which you should say yes) and it will install to your local library. When you build the lesson, {renv} will check that the specific package versions are installed and if they aren’t, it will install them for you.

Carpentries-Specific Packages

We have created three R packages that were designed to work explicitly with our lessons. These should be automatically installed by a special configuration file inside of the lesson repository.

Lesson Template ({sandpaper})

The {sandpaper} package creates and curates the lesson template. It is the only package that the maintainers of the lesson template need to interface with. The majority of maintainers need only one function: build_lesson(), which will evaluate all new content and render it to an HTML page on their local machine.

[i] Key Packages

The {sandpaper} package relies heavily on a few packages internally to make sure that we do not reinvent the wheel too much and that we gain the benefits from their tests:

• {pkgdown} provides a scaffolding for us to be able to wrap our HTML (translated from markdown) and metadata in a framework that can live separately from the lesson template.
• {knitr} is the engine that we use to translate RMarkdown to Markdown without styling the output document.
• {rmarkdown} provides functions that give us access to pandoc. At the moment, the only function we use from RMarkdown is pandoc_convert() to transform markdown to HTML.
• {usethis} provides user interface functions for creating projects and working with github.
• {gert} gives users access to git from R without the need for having it installed.

Why RMarkdown?

We have decided on using the RMarkdown format because it is a VERY powerful tool that allows you to write code and text without having to copy and paste output/images/tables. It has support for several languages including python, BASH, and SQL. Most importantly: the code rendering features of it are purely optional; you can strip away all R-associated content and be left with a markdown document.

This is a system that has already been used for our R lessons for several years and works fairly well.

The Two-Step: Building Locally

To preview the lessons, we build the static files locally, without any servers. While it is possible to go directly from RMarkdown to HTML, we use a two-step process:

1. Generate static markdown from the source files with a hash of the original file in the YAML header
2. Using external style template, generate website HTML from the built markdown files

Now, the first reaction you may have is, “Why do we need a two step process when we can generate HTML directly from RMarkdown?” The answer is three-fold.

First, if there is R code in the RMarkdown document, it takes time to execute the code and add it in to the output document. By having an intermediate markdown document that contains a hash of the original document, we can make sure that we only build the files that have changed in content with minimal overhead in a manner that is compatible with RMarkdown’s chunk caching feature.

Second, if any generated element of your page creates invalid HTML, it usually causes a cascading effect that can be hard to track down if you are looking at raw HTML output. Having a markdown intermediate allows us to more easily single out any potential problems. This particular reason is important in the next section for scheduled deployments where we avoid problems raised explicitly by checking the output.

Third, having the rendered markdown documents makes them easy to use with other HTML or pandoc templates not provided by The Carpentries.

You might also be wondering “why aren’t the rendered markdown and HTML files being tracked by git?” The answer is that, they will be, just not on your computer.

[i] Deployment (GitHub)

Because some lesson content will be auto-generated from an ecosystem of software that is constantly evolving, it’s important to make sure a few things are true:

1. The website is always up and available (barring blackouts from GitHub)
2. The style of the lessons can be updated on the fly.
3. The output of the lessons match what the maintainers expect.
4. Changes in the lesson output is easily auditable via git diff.

Because we have two sources of entropy (software environment needed to build the lesson content and CSS/HTML/JS needed to create the website), it’s important to separate these steps because if there is a failure, having them separate allows us to audit the failure and fix the issue on the appropriate end. Thus, we use the same two-step system we presented earlier in local rendering, with weekly and monthly checks:

Zooming in on the Pull request workflow we can see that there is only one point of entry. The box elements represent user interactions, the ellipses are GitHub workflows. The color indicates permission levels for the workflows. Lavender workflows only have the ability to check out code and compare it. Wheat/yellow colored workflows have full permissions to push to the repository:

Because we only track the source files of the lesson and not the output from the maintainer’s computer, we need to rely on Continuous Integration to rebuild the lesson and deploy it to the cloud. We still use the two-step process here, but the difference is that we don’t want to make any extra commits to the main branch, so instead of creating the commits in two extra directories, we create them in orphan branches called md-sources and gh-pages. The latter is familiar to most lesson maintainers, and the former serves as a staging and evaluation area for changes in generated content.

Takeaway from this diagram: Yes, this is complex, but its modularity ensures that we can replace components without needing to significantly modify any of the other components.

The diagram above describes the dependency graph of the lesson template. Each component in the graph represents either a file (beige files/folders/documents) or tools (boxes). The edges (lines connecting the components) are categorized with the following labels:

depends

(e.g. a -> b means component a depends on component b). This is usually reserved for lesson artifacts such as generated markdown and HTML. If a component is dependent on another component, that means that it can not exist without the upstream component. Changes in the upstream component can happen independently to the downstream component, but these changes will affect the state of the downstream component.

uses

(e.g. a -> b means component a uses aspects of component b). In this relationship, each of these components are independent and they can change independent of one another without necessarily affecting the state of the other.

contains

(e.g. a -> b means component a contains component b). A component that contains another component must change when the contained component changes. For example, any changes to the package cache lockfile requires a corresponding commit in the source files.

modifies

(e.g. a -> b means component a modifies component b). Modification occurs when a component is the agent of change for a downstream component, but changes in the upstream component do not necessarily correspond to a change in the downstream component. For example the GitHub Actions component is the only one on this graph that can modify other components (including itself), but if it were to undergo a change to update frequency, that would not necessarily affect the changes that the downstream components see.

Structural Elements

The first issue is common for all websites and has several available solutions that exits in several languages.

At the moment, The Carpentries website supports i18n in a low-rent manner by including a yaml dictionary in the _data directory and switching languages via the site.data.language variable (as shown in this example that translates “This content is open source”). Though, how exactly this is accessible via the main site is not clear.

To make sure that the translations are compatible no matter what tooling we use (R, Python, JavaScript, PhP), we should store the translations in the *.po (portable object) so that each language can use its own gettext() utility to swap out the translations.

Because it will be associated with the lesson template itself, the *po files will live in the {varnish} package and be used from R to translate messages when the website is being generated.

References and Definitions

References for definitions is achieved via the {glosario} project where the glossary is formatted as a yaml file and there are python, and R libraries that can be used to extract specific translations for these glossaries.

Page Content (prose)

This is a topic that is currently not well addressed and is quite hard to do because translating prose is much harder than translating individual messages because the context of an individual paragraph in a section is important. David Pérez-Suárez has proposed to use a {gettext} solution because this is a standard for translating messages in several computer programs. He found a python + BASH project called po4gitbook that will convert markdown content to po files for translation and back again. However, he’s finding that it breaks down a lot with parsing markdown elements like lists and R chunks. I’m thinking that a solution is to use parse the markdown with the commonmark XML spec and then use that to extract the paragraph elements, recast them into markdown and use those for basis of the translated messages. This way, parsing won’t be an issue. The big challenge is that the library has to be re-written for that to happen.