From ff39e15fe44b6224753ff3c0a0d215941ed008c5 Mon Sep 17 00:00:00 2001 From: Sarthakmistry Date: Mon, 13 Jul 2026 06:10:15 -0400 Subject: [PATCH 01/12] Changes to developer environment and a section for R in documentation. --- docs/examples_r.rst | 97 +++++++++++++ docs/get_started_r.rst | 51 +++++++ docs/git_workflow.rst | 54 ++++++++ docs/index.rst | 8 ++ docs/tox_usage.rst | 303 ++++++++++++++++++++++++++++++++++++++++- openbt_pypkg/tox.ini | 4 + 6 files changed, 514 insertions(+), 3 deletions(-) create mode 100644 docs/examples_r.rst create mode 100644 docs/get_started_r.rst diff --git a/docs/examples_r.rst b/docs/examples_r.rst new file mode 100644 index 0000000..072d839 --- /dev/null +++ b/docs/examples_r.rst @@ -0,0 +1,97 @@ +Examples +======== +.. _Branin: https://www.sfu.ca/~ssurjano/branin.html + +To use |openbt| in R, install the ``Ropenbt`` front-end R interface as +described in :doc:`get_started_r`, then let's create a test function. A +popular one is the Branin_ function: + +.. code-block:: r + + # Test Branin function, rescaled + braninsc <- function(xx) + { + x1 <- xx[1] + x2 <- xx[2] + + x1bar <- 15*x1 - 5 + x2bar <- 15 * x2 + + term1 <- x2bar - 5.1*x1bar^2/(4*pi^2) + 5*x1bar/pi - 6 + term2 <- (10 - 10/(8*pi)) * cos(x1bar) + + y <- (term1^2 + term2 - 44.81) / 51.95 + return(y) + } + + + # Simulate branin data for testing + set.seed(99) + n=500 + p=2 + x = matrix(runif(n*p),ncol=p) + y=rep(0,n) + for(i in 1:n) y[i] = braninsc(x[i,]) + +And then we can load the ``Ropenbt`` package and fit a BART model. Here we set +the model type as ``model="bart"`` which ensures we fit a homoscedastic BART +model. The number of MPI threads to use is specified as ``tc=4``. For a list +of all optional parameters, see ``args(openbt)``. + +.. code-block:: r + + library(Ropenbt) + fit=openbt(x,y,tc=4,model="bart",modelname="branin") + +Next we can construct predictions and make a simple plot. Here, we are +calculating the in-sample predictions since we passed the same ``x`` matrix to +the ``predict.openbt()`` function. + +.. code-block:: r + + # Calculate in-sample predictions + fitp=predict.openbt(fit,x,tc=4) + + # Make a simple plot + plot(y,fitp$mmean,xlab="observed",ylab="fitted") + abline(0,1) + +To save the model, use the ``openbt.save()`` function. Similarly, load the +model using ``openbt.load()``. Because the posterior can be large in +sample-based models such as these, the fitted model is saved in a compressed +file format with the extension ``.obt``. + +.. code-block:: r + + # Save fitted model as test.obt in the working directory + openbt.save(fit,"test") + + # Load fitted model to a new object. + fit2=openbt.load("test") + +The standard variable activity information, calculated as the proportion of +splitting rules involving each variable, can be computed using the +``vartivity.openbt()`` function. + +.. code-block:: r + + # Calculate variable activity information + fitv=vartivity.openbt(fit2) + + # Plot variable activity + plot(fitv) + +A more accurate alternative is to calculate the Sobol' indices. + +.. code-block:: r + + # Calculate Sobol indices + fits=sobol.openbt(fit2) + fits$msi + fits$mtsi + fits$msij + +The ``Ropenbt`` package does not currently ship a dedicated automated test +suite of its own; the steps above (fitting the Branin function and checking +that predictions track the observed values) are a reasonable smoke test that +your installation is working end to end. \ No newline at end of file diff --git a/docs/get_started_r.rst b/docs/get_started_r.rst new file mode 100644 index 0000000..a46e009 --- /dev/null +++ b/docs/get_started_r.rst @@ -0,0 +1,51 @@ +Getting Started with R +======================= +.. _Meson: https://mesonbuild.com +.. _ninja: https://ninja-build.org +.. _remotes: https://remotes.r-lib.org + +Installed versions of the |openbt| R package, ``Ropenbt``, provide a front-end +R interface that wraps a dedicated set of |openbt| C++ command line tools. + + +Unlike the |openbt| Python package, ``Ropenbt`` does not build or invoke +Meson_ itself. It is a pure R package with no compiled code of its own; it +simply locates and calls the already-built command line tools (such as +``openbtcli``) on the ``PATH``, or in the current working directory as a +fallback. Building those command line tools is a separate, prerequisite step. + +Build the C++ command line tools +----------------------------------------- +Before installing ``Ropenbt``, you must first build and install the |openbt| +C++ command line tools using Meson_ and ninja_. Follow the +:doc:`get_started_cpp` guide to + +* install the required dependencies (a C++14-compatible compiler, an MPI + installation, and optionally Eigen_), +* install Meson_ and ninja_, and +* build and install the command line tools. + + + +Install Ropenbt +------------------------- +With the command line tools built, install the +``Ropenbt`` R interface directly from Bitbucket using the remotes_ package. First, make sure +``remotes`` is installed: + +.. code-block:: r + + install.packages("remotes") + +Now install ``Ropenbt`` directly from Bitbucket: + +.. code-block:: r + + remotes::install_bitbucket("mpratola/openbt/Ropenbt") + +Note that some ``Ropenbt`` package dependencies may also be installed. Since +``Ropenbt`` itself needs no compilation, this step is quick regardless of +platform. + +See :doc:`examples_r` for a worked example of fitting a model with +``Ropenbt``. \ No newline at end of file diff --git a/docs/git_workflow.rst b/docs/git_workflow.rst index 1a3f40d..92529bb 100644 --- a/docs/git_workflow.rst +++ b/docs/git_workflow.rst @@ -32,3 +32,57 @@ informal git workflow. A minimal set of rules are Developers are encouraged to create PRs early during branch development to begin and record a dialogue with potential reviewers in the PR. + +GitHub Actions +-------------- + +All of the following actions run automatically on every push and pull request to +``main``. A merge should only proceed once all actions pass. + +Documentation +~~~~~~~~~~~~~ + +* **Check Spelling** — Checks all ``.rst`` and ``.md`` files in the repository + for typographic errors using the ``typos`` tool with the ``typos.toml`` + configuration file. + +* **Check Links** — Checks all ``.rst`` and ``.md`` files for broken URLs using + the ``lychee`` tool. In addition to running on push and pull request, this + action runs on a weekly schedule to catch links that break between + contributions. + +* **Build Sphinx Docs** — Builds the |openbt| documentation in both HTML and + PDF format using |tox|. The built documents are uploaded as a downloadable + artifact so that contributors can review rendered documentation without + needing a local build environment. + +Python Package Testing +~~~~~~~~~~~~~~~~~~~~~~ + +* **Test OpenBT Python Source Distribution** — The primary test action. Builds + a Python source distribution and tests it across a matrix of six operating + systems, two MPI implementations (Open MPI and MPICH), and five Python + versions (3.10–3.14). The built source distribution is also uploaded as an + artifact for manual upload to PyPI at release time. This action additionally + runs on published releases. + +* **Test OpenBT Developer-mode Installation** — Tests the editable installation + (``pip install -e .``) on a reduced matrix. MPI is intentionally installed + |via| |pip| rather than a system package manager to confirm that pip-installed + MPI implementations work correctly. + +* **Test OpenBT in Anaconda** — Tests installation inside a conda environment + across six operating systems using a prebuilt Open MPI installed |via| |pip|. + +* **Measure OpenBT Python Coverage** — Runs the full Python test suite with + coverage measurement using |tox| and uploads the raw coverage file, XML + report, and HTML report as artifacts. + +C++ Tools Testing +~~~~~~~~~~~~~~~~~ + +* **Test OpenBT C++ Command Line Tools** — Builds and tests the C++ command + line tools directly across a matrix of six operating systems and two MPI + implementations, independently of the Python package. Prints dynamic library + linkage information for each built binary so that developers can verify the + correct MPI implementation was linked. diff --git a/docs/index.rst b/docs/index.rst index 1633abf..8ef7ce1 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -37,6 +37,14 @@ This package is being developed as part of |band| framework_. get_started_cpp bibliography_cpp +.. toctree:: + :numbered: + :maxdepth: 1 + :caption: R User Guide: + + get_started_r + examples_r + .. toctree:: :numbered: :maxdepth: 1 diff --git a/docs/tox_usage.rst b/docs/tox_usage.rst index 5e0a259..2fac2f8 100644 --- a/docs/tox_usage.rst +++ b/docs/tox_usage.rst @@ -2,7 +2,304 @@ Developer Environment ===================== -.. _tox: https://tox.wiki/en/latest/index.html -.. todo:: - Sarthak to write this +Tox +--- +.. _tox Usage: https://tox.wiki/en/latest/index.html +.. _Oliver Bestwalter: https://youtu.be/PrAyvH-tm8E + +Developers are free to setup whatever environment that they may need to +facilitate their work. However, the |openbt| Python package includes a +`tox Usage`_ setup, which developers can also use to automatically setup and +manage dedicated virtual environments for different predefined development tasks. + +Development with |tox| +~~~~~~~~~~~~~~~~~~~~~~ + +The following is a rough guide to help install |tox| as a command line tool in +a dedicated, minimal virtual environment. |tox| is made available with +no need to manually activate its virtual environment. + +.. note:: + Developers that would like to use |tox| should, at the very least, learn + enough about it that they understand the difference between running ``tox`` + and ``tox -r``. + +.. code-block:: console + + $ cd $HOME/local/venv + $ deactivate + $ /path/to/desired/python --version + $ /path/to/desired/python -m venv $HOME/local/venv/.toxbase + $ ./.toxbase/bin/python -m pip list + $ ./.toxbase/bin/python -m pip install --upgrade pip setuptools + $ ./.toxbase/bin/python -m pip install tox + $ ./.toxbase/bin/python -m pip list + $ ./.toxbase/bin/tox --version + +To avoid having to activate ``.toxbase`` every time we would like to work with +|tox|, we setup |tox| in ``PATH``. Note that developers can use this single +|tox| installation for multiple projects. Please replace ``.bash_profile`` +with the appropriate shell configuration file and tailor the following to your +needs. + +.. code-block:: console + + $ mkdir -p $HOME/local/bin + $ ln -s $HOME/local/venv/.toxbase/bin/tox $HOME/local/bin/tox + $ vi $HOME/.bash_profile + $ . $HOME/.bash_profile + $ which tox + $ tox --version + +No work will be carried out by default with the calls ``tox`` and ``tox -r``. + +The following tasks can be run from within the directory hierarchy that contains +the |openbt| |tox| configuration file ``/path/to/OpenBT/openbt_pypkg/tox.ini``: + +* ``tox -r -e nocoverage`` + + * Execute the full test suite for the |openbt| Python package using the code + installed into Python. + +* ``tox -r -e coverage`` + + * Execute the full test suite for the |openbt| Python package and save + coverage results to a coverage file. + * The test runs the package code in the local clone rather than code installed + into Python so that coverage results are clean and straightforward. + * If the environment variable ``COVERAGE_FILE`` is set, then this is the + coverage file that will be written to. If it is not specified, then the + coverage results are written to ``.coverage_openbt``. + +* ``tox -r -e report`` + + * It is intended that this be run after or with ``coverage``. + * Display a code coverage report for the |openbt| package's full test suite + and generate XML and HTML versions of the report. + * The environment variables ``COVERAGE_XML`` and ``COVERAGE_HTML`` can be + provided to specify the names of the files that the associated reports + should be written to. If ``COVERAGE_XML`` is not specified, the XML report + is written to ``coverage.xml``. If ``COVERAGE_HTML`` is not provided, the + HTML report is written to ``htmlcov``. + +* ``tox -r -e check`` + + * Run several checks on the code to report possible issues. + * No files are altered automatically by this task. + +* ``tox -r -e html`` + + * Generate and serve the |openbt| documentation locally as HTML via a local + server at ``http://127.0.0.1:8000``. The browser reloads automatically + whenever documentation source files are changed. + +* ``tox -r -e pdf`` + + * Generate and render the |openbt| documentation locally as a PDF file. + * Users are responsible for installing ``make`` and a compatible LaTeX + installation for immediate use by |tox|. + +Additionally, you can run any combination of the above such as +``tox -r -e report,coverage``. + +Direct use of |tox| virtual environments +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Many of the |tox| tasks will build the |openbt| binary automatically each time +they are run, which can significantly slow development work. In such cases, a +developer will likely start their work by creating a clean virtual environment +for their task using ``tox -r`` and subsequently load and work in that virtual +environment directly. + +Developers can inspect ``tox.ini`` to see what commands are run by their task +and adapt these for their work. + +The following example shows how to run only a single test case using the +``coverage`` virtual environment setup by |tox|. + +.. code-block:: console + + $ cd /path/to/OpenBT/openbt_pypkg + $ tox -r -e coverage + $ . ./.tox/coverage/bin/activate + $ which python + $ python --version + $ python -m pip list + $ python -m pytest openbt.tests.test_brt + +Note that using the ``coverage`` virtual environment directly can be +particularly useful since the package is installed in editable mode and +therefore facilitates interactive development and testing of the Python code. + +|Tox|'s ``-r`` flag takes a conservative approach by wiping and fully +rebuilding the virtual environment from scratch on every invocation, which +guarantees a clean state but adds overhead each time. For iterative work this +accumulates quickly. The ``html`` environment illustrates how to set up the +environment once and then work flexibly inside it rather than letting |tox| +drive every step. + +When |tox| runs the ``html`` environment it launches ``sphinx-autobuild``, a +live-reload server. A developer who wants to invoke ``sphinx-build`` with +specific flags, rebuild on demand, or simply skip the server overhead can +instead create the environment once and activate it directly: + +.. code-block:: console + + $ cd /path/to/OpenBT/openbt_pypkg + $ tox -r -e html + $ # Press Ctrl+C to stop sphinx-autobuild once the environment is ready. + $ . ./.tox/html/bin/activate + $ which sphinx-build + $ sphinx-build -W -E -b html ../docs ../docs/build_html + +On subsequent documentation iterations only the ``sphinx-build`` command is +needed — the environment is already activated and no |tox| rebuild is required. +Omitting ``-E`` on later runs reuses Sphinx's cached environment and speeds up +incremental builds. The live-reload server can also be started directly from +the activated environment when it is useful: + +.. code-block:: console + + $ sphinx-autobuild -W -b html ../docs ../docs/build_html + +Eigen +----- +.. _Eigen: https://gitlab.com/libeigen/eigen + +Eigen_ is a header-only C++ template library for linear algebra. Being +header-only means there is no compiled library to link against, it is used +purely by including its headers directly into source files. + +Installation +~~~~~~~~~~~~ + +Eigen does not need to be installed manually. The |openbt| Meson build system +handles Eigen automatically in two steps. First, Meson searches for an +existing system-wide Eigen installation discoverable |via| ``pkg-config``. If +found, that installation is used for the build. If not found, Meson falls back +to the ``subprojects/eigen.wrap`` file, which instructs it to download Eigen +5.0.1 automatically from GitLab and use it internally for that build. As a +result, Eigen is always available to the build regardless of whether it is +installed on the system. + +Developers on macOS who prefer to have a system-wide installation can install +Eigen |via| Homebrew: + +.. code-block:: console + + $ brew install eigen + + +Meson Build +----------- +.. _Meson: https://mesonbuild.com +.. _ninja: https://ninja-build.org + +The |openbt| Python package uses the Meson_ build system together with its +ninja_ backend to compile the C++ command line tools during installation. +Meson must be installed and available on ``PATH`` before building the package. +Please refer to :ref:`get_started_cpp:Meson installation` for detailed +installation instructions. + +Build Process with Python +~~~~~~~~~~~~~ + +The Meson build is not invoked directly by developers. It is triggered +automatically when the |openbt| Python package is installed |via| + +.. code-block:: console + + $ cd /path/to/OpenBT/openbt_pypkg + $ python -m pip install . + +or in editable mode |via| + +.. code-block:: console + + $ python -m pip install -e . + +Internally, ``setup.py`` defines a custom ``build_clt`` command that runs the +following three Meson commands sequentially from within the ``cpp/`` directory: + +.. code-block:: console + + $ meson setup --wipe --clearcache --buildtype=release builddir \ + -Dprefix=/path/to/src/openbt -Duse_mpi=true -Dpypkg=true + $ meson compile -v -C builddir + $ meson install --quiet -C builddir + +The ``--wipe`` flag deletes and recreates ``builddir`` before every install, +ensuring a clean compile from scratch. The ``--clearcache`` flag clears +Meson's dependency detection cache, forcing it to re-detect the compiler, MPI, +and Eigen installations. + +Files and Directories Created +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A successful ``pip install`` creates the following files and directories: + +* ``openbt_pypkg/cpp/builddir/`` — Meson's working build directory. Ninja + compiles all C++ source files into object files. This directory is wiped and recreated on + every ``pip install`` and can be deleted safely at any time. + +* ``openbt_pypkg/src/openbt/bin/`` — The eight compiled C++ command line + tools installed by ``meson install``: + + .. code-block:: console + + openbtcli openbtpred openbtmixingwts + openbtmixing openbtmixingpred openbtmopareto + openbtsobol openbtvartivity + + .. note:: + Only ``openbtcli``, ``openbtpred``, and ``openbtmixingwts`` are + included in the distributed package. All eight are compiled and + installed to disk regardless. + +* ``openbt_pypkg/src/openbt/include/eigen3/`` — Eigen headers installed + under the package prefix as a side effect of Eigen's own Meson install + step, regardless of whether Eigen came from the system or the bundled + ``subprojects/eigen.wrap``. + +* ``openbt_pypkg/src/openbt/lib/pkgconfig/eigen3.pc`` — A ``pkg-config`` + file for the installed Eigen, with its ``prefix`` pointing into + ``src/openbt/``. + +* ``openbt_pypkg/src/openbt/_version.py`` — Written by ``setuptools_scm`` + from the current git tag, not by Meson. + +Caching +~~~~~~~ + +There are four caching layers involved in the build, each with different +behaviour on a recompile: + +* ``subprojects/packagecache/`` — Stores downloaded Eigen tarballs + (``eigen-5.0.1.tar.bz2`` and its patch) so that Meson does not re-download + them on every build. ``--clearcache`` does not clear this directory; it + persists intentionally across builds. + +* ``cpp/builddir/`` — Ninja's compile cache of object files. Because + ``meson setup --wipe`` is run on every ``pip install``, this cache is never + reused between installs and is always rebuilt from scratch. + +* ``src/openbt/{bin,include,lib}/`` — The install destination written by + ``meson install``. This is the most problematic caching layer: ``meson + install`` overlays new files onto these directories but never removes + stale ones. If a binary is renamed, a tool is removed from the build, or + Eigen headers change, the old files persist silently. When the build + produces unexpected behaviour, these directories should be deleted manually + before reinstalling: + + .. code-block:: console + + $ rm -rf openbt_pypkg/src/openbt/bin/ + $ rm -rf openbt_pypkg/src/openbt/include/ + $ rm -rf openbt_pypkg/src/openbt/lib/ + +* ``openbt_pypkg/.tox/`` — |tox| virtual environments each contain their own + installed copy of the |openbt| package and compiled binaries. Running + ``tox`` without ``-r`` reuses the existing environment and does not + reinstall |openbt| or rerun the Meson build. Running ``tox -r`` forces a + clean environment rebuild and a full ``pip install`` from scratch. diff --git a/openbt_pypkg/tox.ini b/openbt_pypkg/tox.ini index 60ca6f3..aed5cff 100644 --- a/openbt_pypkg/tox.ini +++ b/openbt_pypkg/tox.ini @@ -54,8 +54,12 @@ deps = sphinx sphinxcontrib-bibtex sphinx_rtd_theme + # The command below is for live-reloading of the documentation during development. Uncomment it if you want to use it. + #sphinx-autobuild commands = sphinx-build -W -E -b html {env:DOC_ROOT} {env:DOC_ROOT}/build_html + # The command below is for live-reloading of the documentation during development. Uncomment it if you want to use it. + #sphinx-autobuild -W -E -b html {env:DOC_ROOT} {env:DOC_ROOT}/build_html [testenv:pdf] description = Generate OpenBT PDF-format documentation From 433ef5d3cdda459a9b789deafa0ae21743524fc2 Mon Sep 17 00:00:00 2001 From: Sarthakmistry Date: Mon, 20 Jul 2026 15:35:03 -0400 Subject: [PATCH 02/12] Simplified instructions in documentation --- docs/get_started_r.rst | 30 ++------- docs/git_workflow.rst | 11 ++-- docs/tox_usage.rst | 135 ++++++++++------------------------------- 3 files changed, 41 insertions(+), 135 deletions(-) diff --git a/docs/get_started_r.rst b/docs/get_started_r.rst index a46e009..37953e1 100644 --- a/docs/get_started_r.rst +++ b/docs/get_started_r.rst @@ -1,47 +1,27 @@ Getting Started with R ======================= -.. _Meson: https://mesonbuild.com -.. _ninja: https://ninja-build.org .. _remotes: https://remotes.r-lib.org Installed versions of the |openbt| R package, ``Ropenbt``, provide a front-end R interface that wraps a dedicated set of |openbt| C++ command line tools. - - -Unlike the |openbt| Python package, ``Ropenbt`` does not build or invoke -Meson_ itself. It is a pure R package with no compiled code of its own; it -simply locates and calls the already-built command line tools (such as -``openbtcli``) on the ``PATH``, or in the current working directory as a -fallback. Building those command line tools is a separate, prerequisite step. - -Build the C++ command line tools ------------------------------------------ -Before installing ``Ropenbt``, you must first build and install the |openbt| -C++ command line tools using Meson_ and ninja_. Follow the -:doc:`get_started_cpp` guide to - -* install the required dependencies (a C++14-compatible compiler, an MPI - installation, and optionally Eigen_), -* install Meson_ and ninja_, and -* build and install the command line tools. - - +To build these tools, follow the :doc:`get_started_cpp` guide to build, install, +and test them before continuing. Install Ropenbt ------------------------- With the command line tools built, install the -``Ropenbt`` R interface directly from Bitbucket using the remotes_ package. First, make sure +``Ropenbt`` R interface directly from GitHub using the remotes_ package. First, make sure ``remotes`` is installed: .. code-block:: r install.packages("remotes") -Now install ``Ropenbt`` directly from Bitbucket: +Now install ``Ropenbt`` directly from the codebase: .. code-block:: r - remotes::install_bitbucket("mpratola/openbt/Ropenbt") + remotes::install_github(“https://gitub.com/bandframework/OpenBT”, subdir=”Ropenbt”) Note that some ``Ropenbt`` package dependencies may also be installed. Since ``Ropenbt`` itself needs no compilation, this step is quick regardless of diff --git a/docs/git_workflow.rst b/docs/git_workflow.rst index ab406d3..4cb4bfc 100644 --- a/docs/git_workflow.rst +++ b/docs/git_workflow.rst @@ -67,9 +67,9 @@ Python Package Testing ~~~~~~~~~~~~~~~~~~~~~~ * **Test OpenBT Python Source Distribution** — The primary test action. Builds - a Python source distribution and tests it across a matrix of six operating - systems, two MPI implementations (Open MPI and MPICH), and five Python - versions (3.10–3.14). The built source distribution is also uploaded as an + a Python source distribution and tests it across a matrix of operating + systems, MPI implementations, and Python versions to validate broad + compatibility. The built source distribution is also uploaded as an artifact for manual upload to PyPI at release time. This action additionally runs on published releases. @@ -79,7 +79,7 @@ Python Package Testing MPI implementations work correctly. * **Test OpenBT in Anaconda** — Tests installation inside a conda environment - across six operating systems using a prebuilt Open MPI installed |via| |pip|. + across a matrix of operating systems using a prebuilt Open MPI installed |via| |pip|. * **Measure OpenBT Python Coverage** — Runs the full Python test suite with coverage measurement using |tox| and uploads the raw coverage file, XML @@ -89,7 +89,6 @@ C++ Tools Testing ~~~~~~~~~~~~~~~~~ * **Test OpenBT C++ Command Line Tools** — Builds and tests the C++ command - line tools directly across a matrix of six operating systems and two MPI - implementations, independently of the Python package. Prints dynamic library + line tools directly across a matrix of operating systems and MPI implementations, independently of the Python package. Prints dynamic library linkage information for each built binary so that developers can verify the correct MPI implementation was linked. diff --git a/docs/tox_usage.rst b/docs/tox_usage.rst index 2fac2f8..9e8e9c8 100644 --- a/docs/tox_usage.rst +++ b/docs/tox_usage.rst @@ -54,54 +54,18 @@ needs. No work will be carried out by default with the calls ``tox`` and ``tox -r``. -The following tasks can be run from within the directory hierarchy that contains -the |openbt| |tox| configuration file ``/path/to/OpenBT/openbt_pypkg/tox.ini``: +Run the following from the directory hierarchy that contains the |openbt| +|tox| configuration file ``/path/to/OpenBT/openbt_pypkg/tox.ini`` to see the +full list of available environments and what each one does: -* ``tox -r -e nocoverage`` - - * Execute the full test suite for the |openbt| Python package using the code - installed into Python. - -* ``tox -r -e coverage`` - - * Execute the full test suite for the |openbt| Python package and save - coverage results to a coverage file. - * The test runs the package code in the local clone rather than code installed - into Python so that coverage results are clean and straightforward. - * If the environment variable ``COVERAGE_FILE`` is set, then this is the - coverage file that will be written to. If it is not specified, then the - coverage results are written to ``.coverage_openbt``. - -* ``tox -r -e report`` - - * It is intended that this be run after or with ``coverage``. - * Display a code coverage report for the |openbt| package's full test suite - and generate XML and HTML versions of the report. - * The environment variables ``COVERAGE_XML`` and ``COVERAGE_HTML`` can be - provided to specify the names of the files that the associated reports - should be written to. If ``COVERAGE_XML`` is not specified, the XML report - is written to ``coverage.xml``. If ``COVERAGE_HTML`` is not provided, the - HTML report is written to ``htmlcov``. - -* ``tox -r -e check`` - - * Run several checks on the code to report possible issues. - * No files are altered automatically by this task. - -* ``tox -r -e html`` - - * Generate and serve the |openbt| documentation locally as HTML via a local - server at ``http://127.0.0.1:8000``. The browser reloads automatically - whenever documentation source files are changed. - -* ``tox -r -e pdf`` +.. code-block:: console - * Generate and render the |openbt| documentation locally as a PDF file. - * Users are responsible for installing ``make`` and a compatible LaTeX - installation for immediate use by |tox|. + $ tox list -v -Additionally, you can run any combination of the above such as -``tox -r -e report,coverage``. +Environments can be combined in a single invocation, e.g. +``tox -r -e report,coverage``. Users needing ``pdf`` should note that |tox| +does not install ``make`` or a LaTeX distribution; those must be installed +separately. Direct use of |tox| virtual environments ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -132,37 +96,18 @@ Note that using the ``coverage`` virtual environment directly can be particularly useful since the package is installed in editable mode and therefore facilitates interactive development and testing of the Python code. -|Tox|'s ``-r`` flag takes a conservative approach by wiping and fully -rebuilding the virtual environment from scratch on every invocation, which -guarantees a clean state but adds overhead each time. For iterative work this -accumulates quickly. The ``html`` environment illustrates how to set up the -environment once and then work flexibly inside it rather than letting |tox| -drive every step. - -When |tox| runs the ``html`` environment it launches ``sphinx-autobuild``, a -live-reload server. A developer who wants to invoke ``sphinx-build`` with -specific flags, rebuild on demand, or simply skip the server overhead can -instead create the environment once and activate it directly: +The ``html`` environment can be activated directly in the same way to rebuild +documentation iteratively without paying the cost of a full |tox| rebuild each +time: .. code-block:: console $ cd /path/to/OpenBT/openbt_pypkg $ tox -r -e html - $ # Press Ctrl+C to stop sphinx-autobuild once the environment is ready. $ . ./.tox/html/bin/activate $ which sphinx-build $ sphinx-build -W -E -b html ../docs ../docs/build_html -On subsequent documentation iterations only the ``sphinx-build`` command is -needed — the environment is already activated and no |tox| rebuild is required. -Omitting ``-E`` on later runs reuses Sphinx's cached environment and speeds up -incremental builds. The live-reload server can also be started directly from -the activated environment when it is useful: - -.. code-block:: console - - $ sphinx-autobuild -W -b html ../docs ../docs/build_html - Eigen ----- .. _Eigen: https://gitlab.com/libeigen/eigen @@ -178,10 +123,10 @@ Eigen does not need to be installed manually. The |openbt| Meson build system handles Eigen automatically in two steps. First, Meson searches for an existing system-wide Eigen installation discoverable |via| ``pkg-config``. If found, that installation is used for the build. If not found, Meson falls back -to the ``subprojects/eigen.wrap`` file, which instructs it to download Eigen -5.0.1 automatically from GitLab and use it internally for that build. As a -result, Eigen is always available to the build regardless of whether it is -installed on the system. +to the ``subprojects/eigen.wrap`` file, which instructs it to download a +pinned Eigen version automatically from GitLab and use it internally for that +build. As a result, Eigen is always available to the build regardless of +whether it is installed on the system. Developers on macOS who prefer to have a system-wide installation can install Eigen |via| Homebrew: @@ -203,7 +148,7 @@ Please refer to :ref:`get_started_cpp:Meson installation` for detailed installation instructions. Build Process with Python -~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~ The Meson build is not invoked directly by developers. It is triggered automatically when the |openbt| Python package is installed |via| @@ -219,20 +164,11 @@ or in editable mode |via| $ python -m pip install -e . -Internally, ``setup.py`` defines a custom ``build_clt`` command that runs the -following three Meson commands sequentially from within the ``cpp/`` directory: - -.. code-block:: console - - $ meson setup --wipe --clearcache --buildtype=release builddir \ - -Dprefix=/path/to/src/openbt -Duse_mpi=true -Dpypkg=true - $ meson compile -v -C builddir - $ meson install --quiet -C builddir - -The ``--wipe`` flag deletes and recreates ``builddir`` before every install, -ensuring a clean compile from scratch. The ``--clearcache`` flag clears -Meson's dependency detection cache, forcing it to re-detect the compiler, MPI, -and Eigen installations. +Internally, ``setup.py`` defines a custom ``build_clt`` command that wipes and +rebuilds ``cpp/builddir`` from scratch on every install, forcing Meson to +re-detect the compiler, MPI, and Eigen installations rather than reusing +stale detection results. Developers who need the exact Meson invocation can +inspect ``build_clt`` in ``setup.py`` directly. Files and Directories Created ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -243,19 +179,10 @@ A successful ``pip install`` creates the following files and directories: compiles all C++ source files into object files. This directory is wiped and recreated on every ``pip install`` and can be deleted safely at any time. -* ``openbt_pypkg/src/openbt/bin/`` — The eight compiled C++ command line - tools installed by ``meson install``: - - .. code-block:: console - - openbtcli openbtpred openbtmixingwts - openbtmixing openbtmixingpred openbtmopareto - openbtsobol openbtvartivity - - .. note:: - Only ``openbtcli``, ``openbtpred``, and ``openbtmixingwts`` are - included in the distributed package. All eight are compiled and - installed to disk regardless. +* ``openbt_pypkg/src/openbt/bin/`` — The compiled C++ command line tools + installed by ``meson install``. Only a subset of these are included in the + distributed package; the rest are still compiled and installed to disk. + See ``cpp/meson.build`` for the current list of built tools. * ``openbt_pypkg/src/openbt/include/eigen3/`` — Eigen headers installed under the package prefix as a side effect of Eigen's own Meson install @@ -275,10 +202,10 @@ Caching There are four caching layers involved in the build, each with different behaviour on a recompile: -* ``subprojects/packagecache/`` — Stores downloaded Eigen tarballs - (``eigen-5.0.1.tar.bz2`` and its patch) so that Meson does not re-download - them on every build. ``--clearcache`` does not clear this directory; it - persists intentionally across builds. +* ``subprojects/packagecache/`` — Stores the downloaded Eigen tarball and its + patch so that Meson does not re-download them on every build. + ``--clearcache`` does not clear this directory; it persists intentionally + across builds. * ``cpp/builddir/`` — Ninja's compile cache of object files. Because ``meson setup --wipe`` is run on every ``pip install``, this cache is never @@ -302,4 +229,4 @@ behaviour on a recompile: installed copy of the |openbt| package and compiled binaries. Running ``tox`` without ``-r`` reuses the existing environment and does not reinstall |openbt| or rerun the Meson build. Running ``tox -r`` forces a - clean environment rebuild and a full ``pip install`` from scratch. + clean environment rebuild and a full ``pip install`` from scratch. \ No newline at end of file From 62a9b57180ce09c4d041d53d16b27e2926979cce Mon Sep 17 00:00:00 2001 From: Sarthak Mistry <56183762+Sarthakmistry@users.noreply.github.com> Date: Mon, 20 Jul 2026 15:43:14 -0400 Subject: [PATCH 03/12] Fix GitHub URL in installation instructions --- docs/get_started_r.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/get_started_r.rst b/docs/get_started_r.rst index 37953e1..e1c569f 100644 --- a/docs/get_started_r.rst +++ b/docs/get_started_r.rst @@ -21,11 +21,11 @@ Now install ``Ropenbt`` directly from the codebase: .. code-block:: r - remotes::install_github(“https://gitub.com/bandframework/OpenBT”, subdir=”Ropenbt”) + remotes::install_github(“https://github.com/bandframework/OpenBT”, subdir=”Ropenbt”) Note that some ``Ropenbt`` package dependencies may also be installed. Since ``Ropenbt`` itself needs no compilation, this step is quick regardless of platform. See :doc:`examples_r` for a worked example of fitting a model with -``Ropenbt``. \ No newline at end of file +``Ropenbt``. From 314f4fc24ca9bb836cbda9a9a44234d04eef7738 Mon Sep 17 00:00:00 2001 From: Sarthakmistry Date: Tue, 21 Jul 2026 12:57:37 -0400 Subject: [PATCH 04/12] changed to single quotes --- docs/get_started_r.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/get_started_r.rst b/docs/get_started_r.rst index e1c569f..600628c 100644 --- a/docs/get_started_r.rst +++ b/docs/get_started_r.rst @@ -21,7 +21,7 @@ Now install ``Ropenbt`` directly from the codebase: .. code-block:: r - remotes::install_github(“https://github.com/bandframework/OpenBT”, subdir=”Ropenbt”) + remotes::install_github('https://github.com/bandframework/OpenBT', subdir='Ropenbt') Note that some ``Ropenbt`` package dependencies may also be installed. Since ``Ropenbt`` itself needs no compilation, this step is quick regardless of From b0f06c25cc5b56df367afa70d0d539973b056bb7 Mon Sep 17 00:00:00 2001 From: Jared O'Neal Date: Fri, 24 Jul 2026 07:55:46 -0500 Subject: [PATCH 05/12] Cleanup landing page and homogenize structure. This branch is addressing the R wrapper issue, so we can remove the note. Sarthak has grown the "tox usage" section into much more, so a more precise name is needed. Since R doesn't have an empty bibliography, C++ shouldn't either. --- docs/{tox_usage.rst => developer_environment.rst} | 0 docs/index.rst | 10 +--------- 2 files changed, 1 insertion(+), 9 deletions(-) rename docs/{tox_usage.rst => developer_environment.rst} (100%) diff --git a/docs/tox_usage.rst b/docs/developer_environment.rst similarity index 100% rename from docs/tox_usage.rst rename to docs/developer_environment.rst diff --git a/docs/index.rst b/docs/index.rst index 71496e6..a23dbd4 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -3,11 +3,9 @@ Welcome to |openbt|'s Documentation! .. _Open MPI: https://www.open-mpi.org .. _MPICH: https://www.mpich.org .. _framework: https://bandframework.github.io -.. _Issue 35: https://github.com/bandframework/OpenBT/issues/35 .. _OpenBT repository: https://bitbucket.org/mpratola/openbt/src/master .. _OpenBTMixing repository: https://github.com/jcyannotty/OpenBT - .. image:: images/openbt_logo_rect.png :align: center :alt: OpenBT @@ -37,18 +35,12 @@ frozen. This repository and its contents are being established and developed as part of |band| framework_. -.. note:: - While an R wrapper does exist for the original |openbt| and |openbtmixing| - repositories, that functionality has not yet been included in this new, - combined repository (`Issue 35`_). - .. toctree:: :numbered: :maxdepth: 1 :caption: C++ User Guide: get_started_cpp - bibliography_cpp .. toctree:: :numbered: @@ -75,6 +67,6 @@ This repository and its contents are being established and developed as part of contributing git_workflow documentation - tox_usage + developer_environment versioning release_procedure From 359dd290877c3484a54809f5bf6a901a9c9ccbcf Mon Sep 17 00:00:00 2001 From: Jared O'Neal Date: Fri, 24 Jul 2026 08:01:53 -0500 Subject: [PATCH 06/12] Cleaning as part of PR review --- docs/bibliography_cpp.rst | 13 ------------- docs/index.rst | 2 +- openbt_pypkg/tox.ini | 2 +- 3 files changed, 2 insertions(+), 15 deletions(-) delete mode 100644 docs/bibliography_cpp.rst diff --git a/docs/bibliography_cpp.rst b/docs/bibliography_cpp.rst deleted file mode 100644 index 203aa7f..0000000 --- a/docs/bibliography_cpp.rst +++ /dev/null @@ -1,13 +0,0 @@ -.. raw:: latex - - \cleardoublepage - \begingroup - \renewcommand\chapter[1]{\endgroup} - \phantomsection - -Bibliography -============ - -.. bibliography:: references.bib - :style: plain - :keyprefix: cpp- diff --git a/docs/index.rst b/docs/index.rst index a23dbd4..f9ddb99 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -27,7 +27,7 @@ it can be built with MPI installed on a laptop using the system's package manager or with MPI installations on leadership class platforms and clusters that were installed by experts and optimized for their specific platform. -This repository was established by merging the contents of the original Bitbucket +This project was established by merging the contents of the original Bitbucket `OpenBT repository`_ with the `OpenBTMixing repository`_, which was based off of the former. It, therefore, will supersede those two repositories, which will be frozen. diff --git a/openbt_pypkg/tox.ini b/openbt_pypkg/tox.ini index aed5cff..86a4cb1 100644 --- a/openbt_pypkg/tox.ini +++ b/openbt_pypkg/tox.ini @@ -54,7 +54,7 @@ deps = sphinx sphinxcontrib-bibtex sphinx_rtd_theme - # The command below is for live-reloading of the documentation during development. Uncomment it if you want to use it. + # Uncomment the following dependence if optional live-reloading will be used in this task #sphinx-autobuild commands = sphinx-build -W -E -b html {env:DOC_ROOT} {env:DOC_ROOT}/build_html From 5b5a86ef971a6fc8bc43e379da6c02bf1985b48d Mon Sep 17 00:00:00 2001 From: Jared O'Neal Date: Fri, 24 Jul 2026 11:12:18 -0500 Subject: [PATCH 07/12] Restructure the dev env section. Since we ask users to refer to the C++ CLT build script for more information, I moved some technical information from this section to the script's docs. This makes sense since we shouldn't ask users to read the dev guide. Split up the intermediate/cache information so that info only appears in the section related to the tools that create them. Will reevaluate with Sarthak to determine if this is helpful. Having this section only point developers to the "list" tox subcommand is lovely. Improved the tox.ini descriptions to improve the self-documentation contained in that file. --- docs/developer_environment.rst | 264 ++++++++++++++++----------------- openbt_pypkg/tox.ini | 12 +- tools/build_openbt_clt.sh | 18 +++ 3 files changed, 157 insertions(+), 137 deletions(-) diff --git a/docs/developer_environment.rst b/docs/developer_environment.rst index 9e8e9c8..a997662 100644 --- a/docs/developer_environment.rst +++ b/docs/developer_environment.rst @@ -3,15 +3,104 @@ Developer Environment ===================== +This section is a repository of information that might be potentially useful to +developers. Note that information regarding intermediate files/caches that are +created automatically, which might cause issues during development and testing, +is split across sections. + +Eigen +----- +.. _Eigen: https://gitlab.com/libeigen/eigen + +Eigen_ is a header-only C++ template library for linear algebra. Being +header-only means there is no compiled library to link against, it is used +purely by including its headers directly into source files. + +Installation +~~~~~~~~~~~~ + +The |openbt| Meson build system satisfies the Eigen dependence automatically. +First, Meson uses different techniques to search for an existing Eigen +installation. If found, that installation is used for the build. If not found, +Meson falls back to the ``subprojects/eigen.wrap`` file, which instructs it to +download a pinned Eigen version automatically from Eigen's repository and use it +internally for that build. As a result, Eigen is always available to the build +regardless of whether it is preinstalled on the system. + +Developers using macOS who need to test the build system or who prefer to have a +system-wide installation can install Eigen |via| Homebrew: + +.. code-block:: console + + $ brew install eigen + +Meson Build +----------- +.. _Meson: https://mesonbuild.com +.. _ninja: https://ninja-build.org + +The |openbt| Python package uses the Meson_ build system together with its +ninja_ backend to compile the C++ command line tools during installation. +Please refer to the relevant installation instructions to determine if manual +installation of these tools is required for a particular task. + +Please refer to the documentation in ``tools/build_openbt_clt.sh`` script for +information about using that script, for an example of how to configure and use +the Meson build system, and for potential build difficulties (|eg| due to +intermediate and cached files). + +Build Process with Python +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Meson build is not invoked directly by developers working on or testing the +Python package. The build is triggered automatically when the |openbt| Python +package is installed |via| + +.. code-block:: console + + $ cd /path/to/OpenBT/openbt_pypkg + $ python -m pip install . + +or in editable mode |via| + +.. code-block:: console + + $ python -m pip install -e . + +It is also invoked automatically to build wheels. We generally refer to this +automated process as a "package build." + +Internally, ``setup.py`` defines a custom ``build_clt`` command that wipes and +rebuilds the Meson build directory ``openbt_pypkg/cpp/builddir`` from scratch on +every package build, forcing Meson to re-detect the compiler, MPI, and Eigen +installations rather than reusing stale detection results. Developers who need +the exact Meson invocation can inspect ``build_clt`` in ``setup.py`` directly. + +A successful package build creates the following files and directories: + +* ``openbt_pypkg/cpp/builddir/`` — Meson's working build directory. Build + output including object files are stored here. Since this directory is wiped + and recreated on every package build, it can be deleted safely at any time. + +* ``openbt_pypkg/src/openbt/_version.py`` — Written by ``setuptools_scm`` + from the current git tag, not by Meson. + +Note that while ``openbt_pypkg/cpp`` officially contains the package's C++ +source code and Meson build system, its contents simply alias the actual code +and build system defined at the root of the repository. Therefore, for example, +all intermediate and cached issues associated with the base folder also exist +for package builds. + Tox --- -.. _tox Usage: https://tox.wiki/en/latest/index.html -.. _Oliver Bestwalter: https://youtu.be/PrAyvH-tm8E +.. _tox setup: https://tox.wiki/en/latest/index.html Developers are free to setup whatever environment that they may need to -facilitate their work. However, the |openbt| Python package includes a -`tox Usage`_ setup, which developers can also use to automatically setup and -manage dedicated virtual environments for different predefined development tasks. +facilitate their work with the Python package. However, the package includes a +`tox setup`_, which developers can also use to automatically setup and manage +dedicated virtual environments for different predefined development tasks. Some +tasks are more broadly useful at the level of the whole repository since they +can, for instance, build the User Guides for all |openbt| tools. Development with |tox| ~~~~~~~~~~~~~~~~~~~~~~ @@ -23,7 +112,7 @@ no need to manually activate its virtual environment. .. note:: Developers that would like to use |tox| should, at the very least, learn enough about it that they understand the difference between running ``tox`` - and ``tox -r``. + and ``tox -r``. Some potential issues are highlighted below. .. code-block:: console @@ -47,7 +136,7 @@ needs. $ mkdir -p $HOME/local/bin $ ln -s $HOME/local/venv/.toxbase/bin/tox $HOME/local/bin/tox - $ vi $HOME/.bash_profile + $ vi $HOME/.bash_profile (add $HOME/local/bin to PATH) $ . $HOME/.bash_profile $ which tox $ tox --version @@ -62,19 +151,24 @@ full list of available environments and what each one does: $ tox list -v -Environments can be combined in a single invocation, e.g. -``tox -r -e report,coverage``. Users needing ``pdf`` should note that |tox| -does not install ``make`` or a LaTeX distribution; those must be installed -separately. +Two or more tasks can be executed in a single invocation, (|eg| ``tox -r -e +report,coverage``). Users needing ``pdf`` should note that |tox| does not +install ``make`` or a LaTeX distribution; those must be installed separately. + +The |tox| tool caches all of its virtual environments in ``openbt_pypkg/.tox/``. +Running ``tox -r `` forces a clean environment rebuild including +installation of (potentially more modern) dependencies and a full package build +from scratch. Happily, developers can activate and work directly in |tox|'s +cached virtual environments. Direct use of |tox| virtual environments ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Many of the |tox| tasks will build the |openbt| binary automatically each time -they are run, which can significantly slow development work. In such cases, a -developer will likely start their work by creating a clean virtual environment -for their task using ``tox -r`` and subsequently load and work in that virtual -environment directly. +they are run, which can significantly slow development work. In such cases, +developer productivity can benefit from creating a clean virtual environment for +their task using ``tox -r `` and subsequently loading and working in that +virtual environment directly. Developers can inspect ``tox.ini`` to see what commands are run by their task and adapt these for their work. @@ -97,7 +191,7 @@ particularly useful since the package is installed in editable mode and therefore facilitates interactive development and testing of the Python code. The ``html`` environment can be activated directly in the same way to rebuild -documentation iteratively without paying the cost of a full |tox| rebuild each +documentation iteratively without paying the cost of a full package rebuild each time: .. code-block:: console @@ -108,125 +202,29 @@ time: $ which sphinx-build $ sphinx-build -W -E -b html ../docs ../docs/build_html -Eigen ------ -.. _Eigen: https://gitlab.com/libeigen/eigen - -Eigen_ is a header-only C++ template library for linear algebra. Being -header-only means there is no compiled library to link against, it is used -purely by including its headers directly into source files. - -Installation -~~~~~~~~~~~~ - -Eigen does not need to be installed manually. The |openbt| Meson build system -handles Eigen automatically in two steps. First, Meson searches for an -existing system-wide Eigen installation discoverable |via| ``pkg-config``. If -found, that installation is used for the build. If not found, Meson falls back -to the ``subprojects/eigen.wrap`` file, which instructs it to download a -pinned Eigen version automatically from GitLab and use it internally for that -build. As a result, Eigen is always available to the build regardless of -whether it is installed on the system. - -Developers on macOS who prefer to have a system-wide installation can install -Eigen |via| Homebrew: - -.. code-block:: console - - $ brew install eigen - - -Meson Build ------------ -.. _Meson: https://mesonbuild.com -.. _ninja: https://ninja-build.org - -The |openbt| Python package uses the Meson_ build system together with its -ninja_ backend to compile the C++ command line tools during installation. -Meson must be installed and available on ``PATH`` before building the package. -Please refer to :ref:`get_started_cpp:Meson installation` for detailed -installation instructions. - -Build Process with Python -~~~~~~~~~~~~~~~~~~~~~~~~~ - -The Meson build is not invoked directly by developers. It is triggered -automatically when the |openbt| Python package is installed |via| - -.. code-block:: console - - $ cd /path/to/OpenBT/openbt_pypkg - $ python -m pip install . - -or in editable mode |via| - -.. code-block:: console - - $ python -m pip install -e . - -Internally, ``setup.py`` defines a custom ``build_clt`` command that wipes and -rebuilds ``cpp/builddir`` from scratch on every install, forcing Meson to -re-detect the compiler, MPI, and Eigen installations rather than reusing -stale detection results. Developers who need the exact Meson invocation can -inspect ``build_clt`` in ``setup.py`` directly. - -Files and Directories Created -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A successful ``pip install`` creates the following files and directories: - -* ``openbt_pypkg/cpp/builddir/`` — Meson's working build directory. Ninja - compiles all C++ source files into object files. This directory is wiped and recreated on - every ``pip install`` and can be deleted safely at any time. - -* ``openbt_pypkg/src/openbt/bin/`` — The compiled C++ command line tools - installed by ``meson install``. Only a subset of these are included in the - distributed package; the rest are still compiled and installed to disk. - See ``cpp/meson.build`` for the current list of built tools. - -* ``openbt_pypkg/src/openbt/include/eigen3/`` — Eigen headers installed - under the package prefix as a side effect of Eigen's own Meson install - step, regardless of whether Eigen came from the system or the bundled - ``subprojects/eigen.wrap``. - -* ``openbt_pypkg/src/openbt/lib/pkgconfig/eigen3.pc`` — A ``pkg-config`` - file for the installed Eigen, with its ``prefix`` pointing into - ``src/openbt/``. - -* ``openbt_pypkg/src/openbt/_version.py`` — Written by ``setuptools_scm`` - from the current git tag, not by Meson. - Caching ~~~~~~~ - -There are four caching layers involved in the build, each with different -behaviour on a recompile: - -* ``subprojects/packagecache/`` — Stores the downloaded Eigen tarball and its - patch so that Meson does not re-download them on every build. - ``--clearcache`` does not clear this directory; it persists intentionally - across builds. - -* ``cpp/builddir/`` — Ninja's compile cache of object files. Because - ``meson setup --wipe`` is run on every ``pip install``, this cache is never - reused between installs and is always rebuilt from scratch. - -* ``src/openbt/{bin,include,lib}/`` — The install destination written by - ``meson install``. This is the most problematic caching layer: ``meson - install`` overlays new files onto these directories but never removes +Tox tasks that build the |openbt| package in editable mode install build +products, such as the command line tools, directly in a developer's clone rather +than caching them inside the task's ``openbt_pypkg/.tox/`` folder. These +cached files, which can occasionally cause issues, are + +* ``openbt_pypkg/src/openbt/{bin,include,lib}/`` — The install destination + populated by ``meson install``. This is the most problematic caching layer: + ``meson install`` overlays new files onto these directories but never removes stale ones. If a binary is renamed, a tool is removed from the build, or - Eigen headers change, the old files persist silently. When the build - produces unexpected behaviour, these directories should be deleted manually - before reinstalling: - - .. code-block:: console + Eigen headers change, the old files persist silently. Consider deleting these + if the build produces unexpected behaviour. Note that, of these contents, + only a subset of the command line tools in ``bin`` is included in a package + build. See ``meson.build`` for the current list of built tools. - $ rm -rf openbt_pypkg/src/openbt/bin/ - $ rm -rf openbt_pypkg/src/openbt/include/ - $ rm -rf openbt_pypkg/src/openbt/lib/ +* ``openbt_pypkg/src/openbt/include/eigen3/`` — Eigen headers installed + under the package prefix as a side effect of Eigen's own Meson install step, + regardless of whether Eigen came from the system or the bundled + ``subprojects/eigen.wrap``. These files are uninmportant once the command + line tools are built and are not included in package distributions. -* ``openbt_pypkg/.tox/`` — |tox| virtual environments each contain their own - installed copy of the |openbt| package and compiled binaries. Running - ``tox`` without ``-r`` reuses the existing environment and does not - reinstall |openbt| or rerun the Meson build. Running ``tox -r`` forces a - clean environment rebuild and a full ``pip install`` from scratch. \ No newline at end of file +* ``openbt_pypkg/src/openbt/lib/pkgconfig/eigen3.pc`` — A ``pkg-config`` + file for the installed Eigen, with its ``prefix`` pointing into + ``src/openbt/``, that is installed as a side effect. This file is unimportant + and is not included in package distributions. diff --git a/openbt_pypkg/tox.ini b/openbt_pypkg/tox.ini index 86a4cb1..0f9f69e 100644 --- a/openbt_pypkg/tox.ini +++ b/openbt_pypkg/tox.ini @@ -8,7 +8,9 @@ requires = tox>=4 env_list = [testenv] -description = Run OpenBT's full test suite with or without coverage +description = + coverage: Run OpenBT's full test suite with coverage + nocoverage: Run OpenBT's full test suite without coverage passenv = COVERAGE_HTML COVERAGE_XML @@ -28,7 +30,10 @@ commands = coverage: coverage run --rcfile={toxinidir}/.coveragerc --data-file={env:COV_FILE} -m pytest ./src/openbt/tests [testenv:report] -description = Generate XML and HTML format coverage reports +description = Write coverage results to stdout as well as generate XML and HTML + format coverage reports. This is typically run after or at the same time as + the coverage task. See tox.ini for information on env vars that control + where the reports are written. depends = coverage deps = coverage skip_install = true @@ -38,8 +43,7 @@ commands = coverage report --data-file={env:COV_FILE} [testenv:check] -# The work done in this task does not alter any files. -description = Check code against typical Python standards +description = Check code against typical Python standards. This task does not alter any files. deps = setuptools flake8 diff --git a/tools/build_openbt_clt.sh b/tools/build_openbt_clt.sh index b5a58c3..e1292aa 100755 --- a/tools/build_openbt_clt.sh +++ b/tools/build_openbt_clt.sh @@ -12,6 +12,24 @@ # This script returns exit codes that should make it compatible with use in CI # build processes. # +# Intermediate & cached files +# --------------------------- +# This script has Meson create and use the /path/to/OpenBT/builddir folder for +# the build. Developers can use this script to create that folder and then use +# Meson manually with that folder to develop and test the code. Users could +# similarly use the contents of the script to guide custom builds. The Meson +# setup, compile, and install commands in the script might provide a good +# starting point for such efforts. +# +# While the /path/to/OpenBT/subprojects folder does contain necessary files +# under version control, it can also contain cached third-party dependencies +# such as Eigen's source code. The subprojects/packagecache folder can also +# contain cached files such as third-party dependence tarballs and patches. +# Please note that setting up the Meson build directory with the --clearcache +# flag does **not** remove such files. Rather, they intentionally persist +# across builds. Consider reviewing those contents if Meson uses Eigen versions +# or installations different from those intended. +# #####----- HARDCODED VALUES use_mpi=true From 4554a64e63938324ec6d10010ec7a8a7f27481de Mon Sep 17 00:00:00 2001 From: Jared O'Neal Date: Fri, 24 Jul 2026 12:00:12 -0500 Subject: [PATCH 08/12] Cleaning docs as part of PR review. --- docs/examples_r.rst | 18 +++++++----------- docs/get_started_r.rst | 20 ++++++++++++++------ docs/git_workflow.rst | 25 ++++++++++++++----------- 3 files changed, 35 insertions(+), 28 deletions(-) diff --git a/docs/examples_r.rst b/docs/examples_r.rst index 072d839..8ac12f9 100644 --- a/docs/examples_r.rst +++ b/docs/examples_r.rst @@ -2,9 +2,10 @@ Examples ======== .. _Branin: https://www.sfu.ca/~ssurjano/branin.html -To use |openbt| in R, install the ``Ropenbt`` front-end R interface as -described in :doc:`get_started_r`, then let's create a test function. A -popular one is the Branin_ function: +To use |openbt| in R, install ``Ropenbt`` as described in :doc:`get_started_r`. +This example assumes that the command line tools were built with MPI support. + +Let's create a test function. A popular one is the Branin_ function: .. code-block:: r @@ -25,7 +26,7 @@ popular one is the Branin_ function: } - # Simulate branin data for testing + # Simulate Branin data for testing set.seed(99) n=500 p=2 @@ -34,7 +35,7 @@ popular one is the Branin_ function: for(i in 1:n) y[i] = braninsc(x[i,]) And then we can load the ``Ropenbt`` package and fit a BART model. Here we set -the model type as ``model="bart"`` which ensures we fit a homoscedastic BART +the model type as ``model="bart"``, which ensures that we fit a homoscedastic BART model. The number of MPI threads to use is specified as ``tc=4``. For a list of all optional parameters, see ``args(openbt)``. @@ -85,13 +86,8 @@ A more accurate alternative is to calculate the Sobol' indices. .. code-block:: r - # Calculate Sobol indices + # Calculate Sobol' indices fits=sobol.openbt(fit2) fits$msi fits$mtsi fits$msij - -The ``Ropenbt`` package does not currently ship a dedicated automated test -suite of its own; the steps above (fitting the Branin function and checking -that predictions track the observed values) are a reasonable smoke test that -your installation is working end to end. \ No newline at end of file diff --git a/docs/get_started_r.rst b/docs/get_started_r.rst index 600628c..5ccc551 100644 --- a/docs/get_started_r.rst +++ b/docs/get_started_r.rst @@ -2,10 +2,14 @@ Getting Started with R ======================= .. _remotes: https://remotes.r-lib.org -Installed versions of the |openbt| R package, ``Ropenbt``, provide a front-end -R interface that wraps a dedicated set of |openbt| C++ command line tools. -To build these tools, follow the :doc:`get_started_cpp` guide to build, install, -and test them before continuing. +Installed versions of the |openbt| R package, ``Ropenbt``, provide a front-end R +interface that wraps a dedicated set of |openbt| C++ command line tools. The +package locates and calls the already-built command line tools (such as +``openbtcli``) by first searching the folders specified in ``PATH``. If they +are not found, it searches the current working directory as a fallback. + +Follow the :doc:`get_started_cpp` guide to build, install, and test the tools +before continuing. Install Ropenbt ------------------------- @@ -27,5 +31,9 @@ Note that some ``Ropenbt`` package dependencies may also be installed. Since ``Ropenbt`` itself needs no compilation, this step is quick regardless of platform. -See :doc:`examples_r` for a worked example of fitting a model with -``Ropenbt``. +Testing +------- +The ``Ropenbt`` package does not currently ship a dedicated automated test suite +of its own. However, executing the full set of steps detailed in +:doc:`examples_r` is a reasonable smoke test that your installation is working +end to end. diff --git a/docs/git_workflow.rst b/docs/git_workflow.rst index 4cb4bfc..e3931dd 100644 --- a/docs/git_workflow.rst +++ b/docs/git_workflow.rst @@ -1,7 +1,5 @@ Git Workflow ============ -Since we are currently standing this repository up, we are working with an -informal git workflow. A minimal set of rules are .. note:: @@ -10,6 +8,9 @@ informal git workflow. A minimal set of rules are which might result in unwanted side effects. Rather, a gatekeeper should resolve the conflicts in a local clone, merge locally, and push. +Since we are currently standing this repository up, we are working with an +informal git workflow. A minimal set of rules are + #. No one should make direct commits to the ``main`` branch. #. Each addition and change should be made on a dedicated feature branch that is based off of the latest commit on the ``main`` branch. Try to group related @@ -66,29 +67,31 @@ Documentation Python Package Testing ~~~~~~~~~~~~~~~~~~~~~~ -* **Test OpenBT Python Source Distribution** — The primary test action. Builds +* **Test |openbt| Python Source Distribution** — The primary test action. Builds a Python source distribution and tests it across a matrix of operating systems, MPI implementations, and Python versions to validate broad - compatibility. The built source distribution is also uploaded as an - artifact for manual upload to PyPI at release time. This action additionally - runs on published releases. + compatibility. This action additionally runs on published releases so that + the source distribution built and tested by the action, which is stored as an + artifact, can be manually uploaded to PyPI as the official release + distribution. -* **Test OpenBT Developer-mode Installation** — Tests the editable installation +* **Test |openbt| Developer-mode Installation** — Tests the editable installation (``pip install -e .``) on a reduced matrix. MPI is intentionally installed |via| |pip| rather than a system package manager to confirm that pip-installed MPI implementations work correctly. -* **Test OpenBT in Anaconda** — Tests installation inside a conda environment - across a matrix of operating systems using a prebuilt Open MPI installed |via| |pip|. +* **Test |openbt| in Anaconda** — Tests installation inside a conda environment + across a matrix of operating systems and installs |via| |pip| a prebuilt + Open MPI installation included in a Python package. -* **Measure OpenBT Python Coverage** — Runs the full Python test suite with +* **Measure |openbt| Python Coverage** — Runs the full Python test suite with coverage measurement using |tox| and uploads the raw coverage file, XML report, and HTML report as artifacts. C++ Tools Testing ~~~~~~~~~~~~~~~~~ -* **Test OpenBT C++ Command Line Tools** — Builds and tests the C++ command +* **Test |openbt| C++ Command Line Tools** — Builds and tests the C++ command line tools directly across a matrix of operating systems and MPI implementations, independently of the Python package. Prints dynamic library linkage information for each built binary so that developers can verify the correct MPI implementation was linked. From 1e3aaff19da9cdea1da19d3513932f8fb3d890b3 Mon Sep 17 00:00:00 2001 From: Jared O'Neal Date: Fri, 24 Jul 2026 15:18:18 -0500 Subject: [PATCH 09/12] Clean up content as part of PR review. It was important to tie cached files at the root level -- editable installations -- rather than to tox. Apparently sphinx substitutions don't work inside bold environments. --- docs/developer_environment.rst | 54 +++++++++++++++++++--------------- docs/git_workflow.rst | 10 +++---- 2 files changed, 35 insertions(+), 29 deletions(-) diff --git a/docs/developer_environment.rst b/docs/developer_environment.rst index a997662..e26011e 100644 --- a/docs/developer_environment.rst +++ b/docs/developer_environment.rst @@ -91,6 +91,33 @@ and build system defined at the root of the repository. Therefore, for example, all intermediate and cached issues associated with the base folder also exist for package builds. +Editable Python package installations install build products, such as the +command line tools, directly in a developer's clone rather than inside the +Python execution environment (|eg| within the ``site-packages`` folder of a +virtual environment). These cached files, which can occasionally cause issues, +are + +* ``openbt_pypkg/src/openbt/{bin,include,lib}/`` — The install destination + populated by ``meson install``. This is the most problematic caching layer: + ``meson install`` overlays new files onto these directories but never removes + stale ones. If a binary is renamed, a tool is removed from the build, or + Eigen headers change, the old files persist silently. Consider deleting these + if the build produces unexpected behaviour. Note that, of these contents, + only a subset of the command line tools in ``bin`` is included in a package + build. See ``meson.build`` for the current list of built tools. + +* ``openbt_pypkg/src/openbt/include/eigen3/`` — Eigen headers installed + under the package prefix as a side effect of Eigen's own Meson install step, + regardless of whether Eigen came from the system or the bundled + ``subprojects/eigen.wrap``. These files are unimportant once the command + line tools are built and are not included in package distributions. + +* ``openbt_pypkg/src/openbt/lib/pkgconfig/eigen3.pc`` — A ``pkg-config`` + file for the installed Eigen, with its ``prefix`` pointing into + ``src/openbt/``, that is installed as a side effect. This file is unimportant + and is not included in package distributions. + + Tox --- .. _tox setup: https://tox.wiki/en/latest/index.html @@ -204,27 +231,6 @@ time: Caching ~~~~~~~ -Tox tasks that build the |openbt| package in editable mode install build -products, such as the command line tools, directly in a developer's clone rather -than caching them inside the task's ``openbt_pypkg/.tox/`` folder. These -cached files, which can occasionally cause issues, are - -* ``openbt_pypkg/src/openbt/{bin,include,lib}/`` — The install destination - populated by ``meson install``. This is the most problematic caching layer: - ``meson install`` overlays new files onto these directories but never removes - stale ones. If a binary is renamed, a tool is removed from the build, or - Eigen headers change, the old files persist silently. Consider deleting these - if the build produces unexpected behaviour. Note that, of these contents, - only a subset of the command line tools in ``bin`` is included in a package - build. See ``meson.build`` for the current list of built tools. - -* ``openbt_pypkg/src/openbt/include/eigen3/`` — Eigen headers installed - under the package prefix as a side effect of Eigen's own Meson install step, - regardless of whether Eigen came from the system or the bundled - ``subprojects/eigen.wrap``. These files are uninmportant once the command - line tools are built and are not included in package distributions. - -* ``openbt_pypkg/src/openbt/lib/pkgconfig/eigen3.pc`` — A ``pkg-config`` - file for the installed Eigen, with its ``prefix`` pointing into - ``src/openbt/``, that is installed as a side effect. This file is unimportant - and is not included in package distributions. +As noted above, some |tox| tasks build the |openbt| package in editable mode. +They, therefore, can suffer from the potential caching issues mentioned above +for direct editable installations of the package. diff --git a/docs/git_workflow.rst b/docs/git_workflow.rst index e3931dd..032d648 100644 --- a/docs/git_workflow.rst +++ b/docs/git_workflow.rst @@ -67,7 +67,7 @@ Documentation Python Package Testing ~~~~~~~~~~~~~~~~~~~~~~ -* **Test |openbt| Python Source Distribution** — The primary test action. Builds +* **Test** |openbt| **Python Source Distribution** — The primary test action. Builds a Python source distribution and tests it across a matrix of operating systems, MPI implementations, and Python versions to validate broad compatibility. This action additionally runs on published releases so that @@ -75,23 +75,23 @@ Python Package Testing artifact, can be manually uploaded to PyPI as the official release distribution. -* **Test |openbt| Developer-mode Installation** — Tests the editable installation +* **Test** |openbt| **Developer-mode Installation** — Tests the editable installation (``pip install -e .``) on a reduced matrix. MPI is intentionally installed |via| |pip| rather than a system package manager to confirm that pip-installed MPI implementations work correctly. -* **Test |openbt| in Anaconda** — Tests installation inside a conda environment +* **Test** |openbt| **in Anaconda** — Tests installation inside a conda environment across a matrix of operating systems and installs |via| |pip| a prebuilt Open MPI installation included in a Python package. -* **Measure |openbt| Python Coverage** — Runs the full Python test suite with +* **Measure** |openbt| **Python Coverage** — Runs the full Python test suite with coverage measurement using |tox| and uploads the raw coverage file, XML report, and HTML report as artifacts. C++ Tools Testing ~~~~~~~~~~~~~~~~~ -* **Test |openbt| C++ Command Line Tools** — Builds and tests the C++ command +* **Test** |openbt| **C++ Command Line Tools** — Builds and tests the C++ command line tools directly across a matrix of operating systems and MPI implementations, independently of the Python package. Prints dynamic library linkage information for each built binary so that developers can verify the correct MPI implementation was linked. From 7fe9599d8630f3b8be3eee97c952c5b5d957971e Mon Sep 17 00:00:00 2001 From: Sarthakmistry Date: Sun, 26 Jul 2026 19:47:59 -0400 Subject: [PATCH 10/12] Added some untracked files in gitignore --- .gitignore | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 11f91bc..6ff199f 100644 --- a/.gitignore +++ b/.gitignore @@ -34,6 +34,9 @@ openbt_pypkg/coverage.xml openbt_pypkg/htmlcov openbt_pypkg/src/openbt.egg-info openbt_pypkg/src/openbt/_version.py +openbt_pypkg/src/openbt/include/ +openbt_pypkg/src/openbt/lib/ + # Other files -.DS_Store +.DS_Store \ No newline at end of file From f1bca0493c2dcaaf807188b61419aed1032078fd Mon Sep 17 00:00:00 2001 From: Sarthakmistry Date: Sun, 26 Jul 2026 20:27:04 -0400 Subject: [PATCH 11/12] resolved broken commands --- docs/developer_environment.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/developer_environment.rst b/docs/developer_environment.rst index e26011e..6cbb816 100644 --- a/docs/developer_environment.rst +++ b/docs/developer_environment.rst @@ -44,8 +44,8 @@ ninja_ backend to compile the C++ command line tools during installation. Please refer to the relevant installation instructions to determine if manual installation of these tools is required for a particular task. -Please refer to the documentation in ``tools/build_openbt_clt.sh`` script for -information about using that script, for an example of how to configure and use +Please refer to the documentation in ``tools/build_openbt_clt.sh`` for +information about using that tool, for an example of how to configure and use the Meson build system, and for potential build difficulties (|eg| due to intermediate and cached files). @@ -183,7 +183,7 @@ report,coverage``). Users needing ``pdf`` should note that |tox| does not install ``make`` or a LaTeX distribution; those must be installed separately. The |tox| tool caches all of its virtual environments in ``openbt_pypkg/.tox/``. -Running ``tox -r `` forces a clean environment rebuild including +Running ``tox -r -e `` forces a clean environment rebuild including installation of (potentially more modern) dependencies and a full package build from scratch. Happily, developers can activate and work directly in |tox|'s cached virtual environments. @@ -194,7 +194,7 @@ Direct use of |tox| virtual environments Many of the |tox| tasks will build the |openbt| binary automatically each time they are run, which can significantly slow development work. In such cases, developer productivity can benefit from creating a clean virtual environment for -their task using ``tox -r `` and subsequently loading and working in that +their task using ``tox -r -e `` and subsequently loading and working in that virtual environment directly. Developers can inspect ``tox.ini`` to see what commands are run by their task @@ -211,7 +211,7 @@ The following example shows how to run only a single test case using the $ which python $ python --version $ python -m pip list - $ python -m pytest openbt.tests.test_brt + $ python -m pytest --pyargs openbt.tests.test_mixing Note that using the ``coverage`` virtual environment directly can be particularly useful since the package is installed in editable mode and From 3f718dbf9cc8d9492f5e7919e573a8a553bb8279 Mon Sep 17 00:00:00 2001 From: Sarthakmistry Date: Sun, 26 Jul 2026 20:27:52 -0400 Subject: [PATCH 12/12] changed wordings for clarity --- docs/examples_r.rst | 2 +- docs/git_workflow.rst | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/examples_r.rst b/docs/examples_r.rst index 8ac12f9..c78dea0 100644 --- a/docs/examples_r.rst +++ b/docs/examples_r.rst @@ -36,7 +36,7 @@ Let's create a test function. A popular one is the Branin_ function: And then we can load the ``Ropenbt`` package and fit a BART model. Here we set the model type as ``model="bart"``, which ensures that we fit a homoscedastic BART -model. The number of MPI threads to use is specified as ``tc=4``. For a list +model. The number of MPI processes to use is specified as ``tc=4``. For a list of all optional parameters, see ``args(openbt)``. .. code-block:: r diff --git a/docs/git_workflow.rst b/docs/git_workflow.rst index 032d648..60c3fcd 100644 --- a/docs/git_workflow.rst +++ b/docs/git_workflow.rst @@ -50,13 +50,13 @@ All of the following actions run automatically on every push and pull request to Documentation ~~~~~~~~~~~~~ -* **Check Spelling** — Checks all ``.rst`` and ``.md`` files in the repository +* **Check Spelling** — Checks all files in the repository for typographic errors using the ``typos`` tool with the ``typos.toml`` configuration file. * **Check Links** — Checks all ``.rst`` and ``.md`` files for broken URLs using the ``lychee`` tool. In addition to running on push and pull request, this - action runs on a weekly schedule to catch links that break between + action runs on a regular schedule to catch links that break between contributions. * **Build Sphinx Docs** — Builds the |openbt| documentation in both HTML and