feanil/development settings

README.rst

@@ -58,6 +58,14 @@ Tutor also features a `development mode`_ which will also help you modify,
 test, and extend openedx-platform. We recommend this method for all Open edX
 developers.
 
+.. note::
+
+   **Experimental:** We are trialing a new bare-metal development workflow based
+   on a dedicated ``development.py`` settings module that builds directly on
+   ``common.py`` and uses ``local.openedx.io`` domains. It is not yet the
+   recommended default, but you can try it by following `Using the development.py
+   settings`_.
+
 Bare Metal (Advanced)
 =====================
 
@@ -200,6 +208,7 @@ A full list of the MFEs expected to run by default are listed below.
 .. _Learner Home MFE: https://github.com/openedx/frontend-app-learner-dashboard
 .. _Learning MFE: https://github.com/openedx/frontend-app-learning/
 .. _Authoring MFE: https://github.com/openedx/frontend-app-authoring/
+.. _Using the development.py settings: ./docs/how-tos/using_development_settings.rst
 
 Expected MFEs and Default Ports
 -------------------------------

cms/envs/common.py

@@ -276,8 +276,15 @@
 
 ############################# SET PATH INFORMATION #############################
 PROJECT_ROOT = path(__file__).abspath().dirname().dirname()  # /edx-platform/cms
+REPO_ROOT = PROJECT_ROOT.dirname()
+COMMON_ROOT = REPO_ROOT / "common"
+OPENEDX_ROOT = REPO_ROOT / "openedx"
 CMS_ROOT = REPO_ROOT / "cms"  # noqa: F405
 LMS_ROOT = REPO_ROOT / "lms"  # noqa: F405
+ENV_ROOT = REPO_ROOT.dirname()  # virtualenv dir /edx-platform is in
+COURSES_ROOT = ENV_ROOT / "data"
+XMODULE_ROOT = REPO_ROOT / "xmodule"
+MEDIA_ROOT = ENV_ROOT / "media_root"
 
 GITHUB_REPO_ROOT = ENV_ROOT / "data"  # noqa: F405
 
@@ -1139,7 +1146,7 @@ def make_lms_template_path(settings):
     # STORAGE_CLASS='storages.backends.s3boto3.S3Boto3Storage',
     # STORAGE_KWARGS=dict(bucket='video-image-bucket'),
     STORAGE_KWARGS=dict(
-        location=MEDIA_ROOT,  # noqa: F405
+        location=Derived(lambda settings: settings.MEDIA_ROOT),
     ),
     DIRECTORY_PREFIX='video-images/',
     BASE_URL=MEDIA_URL,  # noqa: F405

cms/envs/development.py

@@ -0,0 +1,171 @@
+"""
+This settings file is optimized for local development.  It should work equally well for bare-metal development and for
+running inside of development environments such as tutor.
+
+WARNING: THIS FILE IS EXPERIMENTAL
+These settings are currently in development themselves. They may not work for everyone out of the box.
+More updates, including updated documentation will be added as we get closer to removing devstack.py.
+Breaking changes are likely.
+"""
+
+import hashlib
+import hmac
+
+#Helpers for loading plugins and their settings.
+from edx_django_utils.plugins import add_plugins
+
+from openedx.core.djangoapps.plugins.constants import ProjectType, SettingsType
+from openedx.core.lib.derived import derive_settings
+
+# Use the common file as the starting point.
+# pylint: disable=wildcard-import
+from .common import *  # noqa: F403
+
+DEBUG = True
+
+STORAGES['default']['BACKEND'] = 'django.core.files.storage.FileSystemStorage'  # noqa: F405
+STORAGES['staticfiles']['BACKEND'] = 'openedx.core.storage.DevelopmentStorage'  # noqa: F405
+
+# Disable pipeline compression in development
+PIPELINE['PIPELINE_ENABLED'] = False  # noqa: F405
+
+# Revert to the default set of finders as we don't want the production pipeline
+STATICFILES_FINDERS = [
+    'openedx.core.djangoapps.theming.finders.ThemeFilesFinder',
+    'django.contrib.staticfiles.finders.FileSystemFinder',
+    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
+    'pipeline.finders.PipelineFinder',
+]
+
+# Point STATIC_ROOT at test_root/staticfiles/studio so that WEBPACK_LOADER's STATS_FILE resolves
+# here. STATS_FILE is derived from STATIC_ROOT (see openedx/envs/common.py), and this is the
+# directory that `npm run build-dev` / `npm run watch` write the Studio webpack-stats.json into by
+# default (see webpack.common.config.js, whose staticRootCms falls back to
+# ./test_root/staticfiles/studio when STATIC_ROOT_CMS is unset). The base default of
+# ENV_ROOT/staticfiles/studio points *outside* the repo and does not match where webpack writes,
+# so the loader can't find the stats file.
+#
+# NOTE: This is purely so the webpack stats manifest can be located. You are NOT expected to run
+# collectstatic in development -- with DEBUG=True the staticfiles finders serve assets directly
+# from their source dirs (e.g. the bundles in common/static/bundles). The '/studio' suffix mirrors
+# the production convention (cms/envs/production.py).
+STATIC_ROOT = REPO_ROOT / 'test_root' / 'staticfiles' / 'studio'  # noqa: F405
+
+# Whether to run django-require in debug mode.
+REQUIRE_DEBUG = DEBUG
+
+# Run Celery tasks synchronously in-process so local development needs no message broker or worker.
+# The base default (CELERY_ALWAYS_EAGER = False, openedx/envs/common.py) makes task-enqueuing code
+# paths -- e.g. the event fired on xblock creation -- try to reach a broker and 500 with
+# "Connection refused". This matches the old devstack behavior.
+CELERY_ALWAYS_EAGER = True
+
+LMS_BASE = 'local.openedx.io:8000'
+LMS_ROOT_URL = f'http://{LMS_BASE}'
+
+CMS_BASE = 'studio.local.openedx.io:8001'
+CMS_ROOT_URL = f'http://{CMS_BASE}'
+ALLOWED_HOSTS = ['studio.local.openedx.io']
+
+# Dealing with CORS
+CORS_ALLOW_CREDENTIALS = True
+# Each development MFE is served under apps.local.openedx.io on its own port. In practice the CMS
+# only needs to accept cross-origin requests from the authoring MFE (Studio's frontend); the other
+# MFEs talk to the LMS, not Studio. The rest are listed but commented out -- uncomment an origin if
+# that MFE turns out to need to call Studio APIs directly.
+CORS_ORIGIN_WHITELIST = (
+    "http://apps.local.openedx.io:2001",  # authoring (Studio)
+    # "http://apps.local.openedx.io:1984",  # communications
+    # "http://apps.local.openedx.io:1993",  # ora-grading
+    # "http://apps.local.openedx.io:1994",  # gradebook
+    # "http://apps.local.openedx.io:1995",  # profile
+    # "http://apps.local.openedx.io:1996",  # learner-dashboard
+    # "http://apps.local.openedx.io:1997",  # account
+    # "http://apps.local.openedx.io:1998",  # catalog
+    # "http://apps.local.openedx.io:1999",  # authn
+    # "http://apps.local.openedx.io:2000",  # learning
+    # "http://apps.local.openedx.io:2002",  # discussions
+    # "http://apps.local.openedx.io:2025",  # admin-console
+)
+
+# Unsafe (POST/PUT/DELETE) requests from the authoring MFE undergo Django's CSRF origin check, so
+# the MFE origin must be trusted here or Studio rejects writes with a 403 ("Origin checking
+# failed"). Scoped to the authoring MFE for the same reason as CORS_ORIGIN_WHITELIST above;
+# uncomment another origin if that MFE needs to make write requests to Studio.
+CSRF_TRUSTED_ORIGINS = [
+    "http://apps.local.openedx.io:2001",  # authoring (Studio)
+    # "http://apps.local.openedx.io:1984",  # communications
+    # "http://apps.local.openedx.io:1993",  # ora-grading
+    # "http://apps.local.openedx.io:1994",  # gradebook
+    # "http://apps.local.openedx.io:1995",  # profile
+    # "http://apps.local.openedx.io:1996",  # learner-dashboard
+    # "http://apps.local.openedx.io:1997",  # account
+    # "http://apps.local.openedx.io:1998",  # catalog
+    # "http://apps.local.openedx.io:1999",  # authn
+    # "http://apps.local.openedx.io:2000",  # learning
+    # "http://apps.local.openedx.io:2002",  # discussions
+    # "http://apps.local.openedx.io:2025",  # admin-console
+]
+
+# Cookie Related Settings
+SESSION_COOKIE_DOMAIN = '.local.openedx.io'
+
+# MFE Development URLs
+# This one needs a trailing slash to load correctly right now.
+LEARNER_HOME_MICROFRONTEND_URL = 'http://apps.local.openedx.io:1996/learner-dashboard/'
+# This one explicitly needs to not have a trailing slash because of how it's used to make other
+# urls.
+LEARNING_MICROFRONTEND_URL = "http://apps.local.openedx.io:2000/learning"
+
+# The course-authoring MFE (frontend-app-authoring) now serves Studio's course outline, pages &
+# resources, etc. The base default is None (openedx/envs/common.py), so Studio's course_index view
+# builds a redirect to None and 500s (`get_course_outline_url` -> `redirect(None)`). Point it at
+# the authoring MFE, which runs on port 2001 under /authoring.
+COURSE_AUTHORING_MICROFRONTEND_URL = "http://apps.local.openedx.io:2001/authoring"
+CATALOG_MICROFRONTEND_URL = "http://apps.local.openedx.io:1998/catalog"
+
+#################### Studio Search (Meilisearch) ####################
+# Enable Studio/library content search (the content.search app behind /api/content_search),
+# pointing at a local Meilisearch on :7700. MEILISEARCH_URL is used by the Python backend;
+# MEILISEARCH_PUBLIC_URL is the URL the browser uses to query Meilisearch directly.
+MEILISEARCH_ENABLED = True
+MEILISEARCH_URL = "http://localhost:7700"
+MEILISEARCH_PUBLIC_URL = "http://localhost:7700"
+
+# Namespace Meilisearch indexes so a dev instance doesn't collide with other indexes on a shared
+# Meilisearch. "openedx_" is a sensible default; the base default in cms/envs/common.py is "".
+# TODO: long term this default should move up to cms/envs/common.py, but changing it there is a
+# breaking change for existing deployments (their unprefixed indexes would need a reindex), so we
+# set it here for development only until that migration is handled separately.
+MEILISEARCH_INDEX_PREFIX = "openedx_"
+
+# Meilisearch derives every API key's value as HMAC-SHA256(master_key, key_uid): you choose the
+# UID, and Meilisearch determines the key value. By fixing both the master key and the UID as shared
+# dev constants, the derived key is the same for everyone, so we compute it here -- rather than each
+# developer having to fetch a randomly-generated key from their own Meilisearch and paste a
+# per-person value into this shared file. MEILISEARCH_MASTER_KEY must match the MEILI_MASTER_KEY of
+# whatever Meilisearch instance you run locally.
+#
+# Meilisearch cannot declare a key at boot, so a key with this UID must also be created once in your
+# Meilisearch (this is idempotent -- Meilisearch returns an error if the UID already exists, which
+# you can ignore):
+#
+#   curl -X POST "http://localhost:7700/keys" \
+#     -H "Authorization: Bearer openedx-insecure-meilisearch-master-key" \
+#     -H "Content-Type: application/json" \
+#     --data-binary '{"uid": "a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", "name": "Open edX backend",
+#                     "actions": ["*"], "indexes": ["openedx_*"], "expiresAt": null}'
+#
+# These are insecure, dev-only values.
+MEILISEARCH_MASTER_KEY = "openedx-insecure-meilisearch-master-key"
+MEILISEARCH_API_KEY_UID = "a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d"
+MEILISEARCH_API_KEY = hmac.new(
+    MEILISEARCH_MASTER_KEY.encode(), MEILISEARCH_API_KEY_UID.encode(), hashlib.sha256
+).hexdigest()
+
+#######################################################################################################################
+#### DERIVE ANY DERIVED SETTINGS
+####
+
+derive_settings(__name__)
+add_plugins(__name__, ProjectType.CMS, SettingsType.DEVSTACK)

docs/how-tos/using_development_settings.rst

@@ -0,0 +1,169 @@
+Using the ``development.py`` settings (experimental)
+####################################################
+.. contents::
+
+Overview
+========
+
+This guide describes an **experimental** way to run the LMS and CMS for local
+development. Instead of the legacy ``devstack.py`` settings (which build on top
+of ``production.py``), it uses a dedicated ``development.py`` settings module for
+each service that builds directly on top of the ``common.py`` defaults.
+
+Two things differ from the older bare-metal / devstack instructions:
+
+* **Settings module:** you pass ``--settings=development`` to ``manage.py``
+  instead of relying on ``devstack.py``.
+* **Domains:** services are addressed through ``local.openedx.io`` subdomains
+  (which resolve to ``127.0.0.1``) rather than ``localhost:<port>``. This gives
+  nicer, production-like hostnames and lets cookie, CORS, and CSRF behavior be
+  exercised more realistically across services and MFEs.
+
+.. warning::
+
+   This workflow is under active development and is **not** yet the recommended
+   default. Some steps may change. If you want a supported development
+   environment today, use `Tutor's development mode`_.
+
+Prerequisites
+=============
+
+Follow the **System Dependencies** and initial **Build Steps** from the
+``Bare Metal (Advanced)`` section of the `openedx-platform README`_ (Python
+3.12, Node, MySQL, Mongo, Memcached, a virtualenv, ``npm clean-install``, and
+``pip install -r requirements/edx/development.txt``). The steps below replace
+only the "Run the Platform" portion of those instructions.
+
+Domain names
+============
+
+``local.openedx.io`` and its subdomains resolve to the loopback address
+(``127.0.0.1``), so no web server or proxy is required. This guide uses:
+
+* ``local.openedx.io`` — LMS
+* ``studio.local.openedx.io`` — CMS / Studio
+* ``apps.local.openedx.io`` — Micro-frontends (MFEs)
+
+Database and migrations
+=======================
+
+Studio content search is on by default (backed by `Meilisearch`_), and its index
+is created during ``cms migrate`` by a post-migrate step -- so start Meilisearch
+before migrating: run it at ``http://localhost:7700`` with ``MEILI_MASTER_KEY``
+matching ``MEILISEARCH_MASTER_KEY`` in ``cms/envs/development.py``, and create the
+backend API key once (a ready-to-run ``curl`` is in the "Studio Search" section of
+that file).
+
+Create the databases and run the one-time setup, passing ``--settings=development``::
+
+  python manage.py lms --settings=development migrate
+  python manage.py lms --settings=development migrate --database=student_module_history
+  python manage.py cms --settings=development migrate
+  python manage.py cms --settings=development reindex_studio   # populate the Studio search index
+
+If Meilisearch was not reachable during ``cms migrate``, the search index is not
+created and content indexing later fails with "primary key inference failed";
+re-run ``cms migrate`` with Meilisearch up and then ``reindex_studio`` to fix it.
+
+Build frontend assets
+======================
+
+Build the webpack bundles once (or run the watcher for a live edit/rebuild
+loop)::
+
+  npm run build-dev      # one-time build
+  # or, for an auto-rebuilding dev loop:
+  npm run watch
+
+You do **not** need to run ``collectstatic``. With ``DEBUG = True`` the
+staticfiles finders serve assets directly from their source directories. The
+``development.py`` settings point ``STATIC_ROOT`` at ``test_root/staticfiles``
+only so that the webpack stats manifest (``webpack-stats.json``) can be located.
+
+Run the LMS and CMS
+===================
+
+First, ensure MySQL, Mongo, and Memcached are running. Then start each service
+with the ``development`` settings, bound to its ``local.openedx.io`` host:
+
+Start the LMS::
+
+  python manage.py lms --settings=development runserver local.openedx.io:8000
+
+Start the CMS::
+
+  python manage.py cms --settings=development runserver studio.local.openedx.io:8001
+
+Set up CMS SSO
+==============
+
+Studio authenticates against the LMS via OAuth. Create the worker user and
+OAuth application (as in the bare-metal instructions), using the Studio
+``local.openedx.io`` redirect URI::
+
+  python manage.py lms --settings=development manage_user studio_worker studio_worker@example.com --unusable-password
+  # DO NOT DO THIS IN PRODUCTION. It will make your auth insecure.
+  python manage.py lms --settings=development create_dot_application studio-sso-id studio_worker \
+      --grant-type authorization-code \
+      --skip-authorization \
+      --redirect-uris 'http://studio.local.openedx.io:8001/complete/edx-oauth2/' \
+      --scopes user_id \
+      --client-id 'studio-sso-key' \
+      --client-secret 'studio-sso-secret'
+
+Run the MFEs
+============
+
+Most of the UI now lives in Micro-frontends, which run separately. Each MFE is
+served under ``apps.local.openedx.io`` on its own port. Clone the MFE repo(s)
+you need next to ``openedx-platform``, install dependencies (``npm
+clean-install``), and start each one with its ``dev`` script. That script points
+``MFE_CONFIG_API_URL`` at the LMS MFE Config API
+(``http://local.openedx.io:8000/api/mfe_config/v1``), so the MFE fetches its
+runtime configuration (LMS/Studio URLs, etc.) from the running LMS::
+
+  npm run dev
+
+At a minimum you will want the Authoring, Learning, and Learner Home MFEs. The
+``development.py`` settings already configure URLs for the full default set:
+
+.. list-table::
+   :header-rows: 1
+
+   * - MFE
+     - Location
+     - Setting
+   * - frontend-app-learning
+     - apps.local.openedx.io:2000/learning
+     - ``LEARNING_MICROFRONTEND_URL``
+   * - frontend-app-authoring
+     - apps.local.openedx.io:2001/authoring
+     - ``COURSE_AUTHORING_MICROFRONTEND_URL``
+   * - frontend-app-learner-dashboard
+     - apps.local.openedx.io:1996/learner-dashboard
+     - ``LEARNER_HOME_MICROFRONTEND_URL``
+   * - frontend-app-account
+     - apps.local.openedx.io:1997/account
+     - ``ACCOUNT_MICROFRONTEND_URL``
+   * - frontend-app-profile
+     - apps.local.openedx.io:1995/profile
+     - ``PROFILE_MICROFRONTEND_URL``
+
+The remaining default MFE URLs (authn, discussions, communications,
+ora-grading, gradebook, catalog, admin-console) are also set in
+``lms/envs/development.py``; see that file for the complete list and ports.
+
+Notes and differences from devstack
+===================================
+
+* **No broker required.** ``CELERY_ALWAYS_EAGER = True`` runs Celery tasks
+  in-process, so you do not need to run a message broker or worker.
+* **MFE configuration is served by the LMS.** The MFE Config API
+  (``/api/mfe_config/v1``) is enabled and populated so MFEs pick up the
+  ``local.openedx.io`` URLs instead of their built-in ``localhost`` defaults.
+* **CORS / CSRF / login redirects** for the default MFE origins are pre-declared
+  in the ``development.py`` files.
+
+.. _Tutor's development mode: https://docs.tutor.edly.io/dev.html
+.. _openedx-platform README: https://github.com/openedx/edx-platform/blob/master/README.rst
+.. _Meilisearch: https://www.meilisearch.com/