edstr

edstr extracts structured variables from unstructured French clinical free text stored in an EDS (Entrepot de Donnees de Sante, i.e. an institutional clinical data warehouse). It provides a pipeline that imports data from an Oracle database, cleans text with regex rules, tokenises and matches user-defined concepts, filters false positives, and exports results as Excel, JSON, and RDS files.

Installation

edstr is not on CRAN. Install from GitHub:

#install.packages("pak")
pak::pak("hebstr/edstr")

System dependencies

• Java JDK (>= 8): required by rJava and DatabaseConnector for Oracle JDBC connectivity. On Debian/Ubuntu: sudo apt install default-jdk then sudo R CMD javareconf.
• Oracle JDBC driver: required for edstr_import() to connect to an Oracle database. Fetch the JAR with DatabaseConnector::downloadJdbcDrivers("oracle") and point the db.path entry of your connection YAML at the folder holding it.
• Quarto command line tool: required to build the vignettes when installing from source. Not needed to use the package.

If you do not need the database import step (edstr_import()), you can still use the cleaning and extraction functions on locally loaded data frames.

Pipeline overview

edstr_config()
      |
      v
edstr_import()
      |
      v
edstr_clean()
      |
      +---------> edstr_view()
      |           (interactive, no save)
      v
edstr_extract()
      |
      v
  .xlsx / .json / .rds

Each step (except edstr_view()) caches its output: edstr_import() and edstr_clean() save Parquet files; edstr_extract() saves an RDS file. If the file already exists, the caching system either loads it, overwrites it, or prompts the user, depending on the edstr_overwrite option.

edstr_view() branches off from cleaned data for interactive pattern exploration and does not save anything.

Exported functions

Function	Description
edstr_config()	Set global options: output directory, file prefix, text column, caching behaviour, and connection file path. Must be called first.
edstr_import()	Execute a SQL query against an Oracle database and cache the result as Parquet.
edstr_clean()	Apply sequential regex replacements to a text column and cache the result.
edstr_extract()	Tokenize text, match concepts, filter false positives, re-match against source text, and export results as XLSX, JSON, and RDS.
edstr_view()	Interactively search for a regex pattern in text and display match frequencies. Does not save.

Quick start

Configure the pipeline

library(edstr)

edstr_config(
  edstr_dirname = "output/my_study",
  edstr_filename = "my_study",
  edstr_text = "note_text",
  edstr_overwrite = TRUE
)

Import and clean

df_import <- edstr_import(
  query = "sql/my_query.sql",
  head = 1000,
  user = "my_user"
)

df_clean <- edstr_clean(
  data = df_import,
  replace = c("\\p{Zs}{2,}" = " ", "\\n" = " ")
)

Explore patterns

Use edstr_view() to iterate on regex patterns before extraction.

edstr_view(
  data = df_clean,
  pattern = "fractur",
  ngrams = 3
)

Extract structured variables

Flat concepts: a named character vector where each element is an independent concept:

result <- edstr_extract(
  data = df_clean,
  concepts = c(fracture = "fractur", femur = "femur|fesf"),
  ngram_max = 2,
  group = "id_pat"
)

Nested concepts: a named list grouping sub-concepts under a root:

result <- edstr_extract(
  data = df_clean,
  concepts = list(
    fracture = list(
      fesf = "fesf|extremite superieure",
      col = "col (du )?femur"
    )
  ),
  group = "id_pat",
  exclus_manual = "ancienne fracture|fracture ouverte"
)

Key features

Concept matching. Concepts are named regex patterns defining clinical entities. A named character vector creates independent concepts; a nested named list groups sub-concepts under a root.

Collapse and intersect modes. collapse = TRUE OR-combines patterns into a single regex: one per root concept as soon as at least one root holds several patterns, otherwise one regex named concepts for the whole set, which drops the root names. intersect = TRUE keeps only documents matching all root-level concepts.

False-positive filtering. Manual exclusions via a user-supplied regex (exclus_manual) and automatic heuristics on long tokens (exclus_auto_token_min).

Source re-matching. After token-level matching on ASCII-transliterated n-grams, matches are located on a transliterated copy of the source and sliced from the original, so extracted text keeps its accents, ligatures and case. Tokens that cannot be confirmed in the source are flagged as mismatches for review.

Pseudonymisation. ano_hash and ano_hide take regex patterns matched case-insensitively against column names, and transform the matching columns before extraction, so every output carries the transformed values: the returned frames, the Excel sheets and the gt tables. Both abort rather than act silently when a pattern matches no column, matches the identifier or group column, or matches the text column.

Built-in caching. edstr_import() and edstr_clean() write Parquet files; edstr_extract() writes an RDS file. The edstr_overwrite option (TRUE / FALSE / NULL) controls whether existing files are overwritten, loaded silently, or trigger an interactive prompt.

Known limitations

• Auto-exclusion in edstr_extract() scans every distinct matched text against every row whose n-gram size exceeds exclus_auto_token_min, three times over (start, end, and start-end anchors). The scan runs whatever exclus_manual is set to, so its cost is not opt-in once any n-gram clears that threshold; exclus_auto_escape shrinks both the rows it scans and the distinct texts scanned against them. Watch for slowdowns on large corpora.
• At the default exclus_auto_token_min = 10, auto-exclusions only apply to n-grams larger than the threshold and no realistic ngram_max value reaches 11, so nothing is scanned and the heuristic never fires. Lower the threshold to enable it.
• ano_hash is pseudonymisation, not anonymisation. The hash is unsalted and stable across runs, which is what makes a patient traceable between two extractions; it also means a small identifier space can be reversed by enumerating it.
• Neither ano_hash nor ano_hide touches the clinical text. data$extract, data$note and the highlighted Excel output carry the source column unredacted, since showing that text is what they are for, and the Parquet caches written upstream by edstr_import() and edstr_clean() keep the source in clear.

Output

edstr_extract() returns a nested list and saves three files:

File	Contents
.xlsx	Excel workbook with one sheet per result type: the extraction, the token stage (patterns, matches, counts, exclusions), concept counts, the source-matching stage (replacements, patterns, matches, counts), unmatched, mismatched, and the call parameters
.json	Summary tables (by token, by concept) and the call parameters (JSON format)
.rds	Full nested list with all intermediate objects (R-native, used for caching)

Articles

Detailed documentation is available as articles on the package website:

• Get started
• Pipeline configuration
• Data import
• Text cleaning
• Text extraction
• Interactive exploration
• Matching: tokens, accents, and the source text

Contributing

Bug reports and feature requests: https://github.com/hebstr/edstr/issues

Source code: https://github.com/hebstr/edstr

License

GPL-3