Skip to content
4 changes: 4 additions & 0 deletions doc/api/warnings.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@
These warning classes can be silenced or filtered via
[`warnings.filterwarnings()`][warnings.filterwarnings].

::: linopy.EmptiedRowWarning
options:
heading_level: 3

::: linopy.EvolvingAPIWarning
options:
heading_level: 3
Expand Down
3 changes: 2 additions & 1 deletion doc/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ linopy 1.0 makes the strict arithmetic convention the only behaviour and puts de
- `Model.from_spec` builds faster: it reads a shifted axis by position, checks a relation's labels against the hash table of their dimension, decides a `where` early when a condition holds nowhere, and sets a scalar scaling factor without broadcasting it. ([\#1169](https://github.com/PyPSA/linopy/pull/1169))
- `Model.from_spec(..., rename=...)` builds each variable and constraint under the name a mapping or a function gives it, e.g. `Generator-p` for the spec's `Generator_p`. Pass the same `rename` to `linopy.spec.evaluate`. ([\#1160](https://github.com/PyPSA/linopy/pull/1160))
- A missing row takes the value that `missing:` sets, e.g. `missing: 0`, in bounds, expressions and conditions alike. A parameter's `where:` limits its coordinates, and a dimension's `one_of:` its labels. Built objects follow the name order of their dimensions, as mathspec sorts every `dims:` list. ([\#1137](https://github.com/PyPSA/linopy/pull/1137))
- Limits: a constraint row that absence leaves without terms raises, where mathspec skips it ([\#993](https://github.com/PyPSA/linopy/issues/993)). `store=True` stores only relations with one key and one value column. A product of two variables in a constraint is accepted only where the data leaves it no live row.
- Limits: a constraint row that the data leaves without terms raises where it cannot hold, where mathspec skips it ([\#993](https://github.com/PyPSA/linopy/issues/993)). `store=True` stores only relations with one key and one value column. A product of two variables in a constraint is accepted only where the data leaves it no live row.

**Other features**

Expand Down Expand Up @@ -131,6 +131,7 @@ linopy 1.0 makes the strict arithmetic convention the only behaviour and puts de
- `Model.from_spec` builds a linear objective when the data sets all quadratic coefficients to zero. ([\#1141](https://github.com/PyPSA/linopy/pull/1141))
- `Model.from_spec` no longer refuses an infinite parameter times a variable absent there, where no row is built. ([\#1143](https://github.com/PyPSA/linopy/pull/1143))
- `Model.from_spec` adds no constraint where every variable of it is masked, where it raised that both sides are constant. ([\#1154](https://github.com/PyPSA/linopy/pull/1154))
- `Model.from_spec` checks a constraint row that the data leaves without a variable term, e.g. through a shift's edge, a sum of absent terms or a zero coefficient: it drops the row with a [`linopy.EmptiedRowWarning`][linopy.EmptiedRowWarning] where it holds and refuses it where it cannot, e.g. `0 == 5`, where it refused both or built `0 * x >= 5`. ([\#1164](https://github.com/PyPSA/linopy/pull/1164))
- `LinearExpression(data, model)` no longer raises on a `.sel` over full coordinates. ([\#1144](https://github.com/PyPSA/linopy/pull/1144))
- `LinearExpression.from_constant` builds on the model's store, so a sparse model's constant stays sparse and adding it emits no `PerformanceWarning`. ([\#1145](https://github.com/PyPSA/linopy/pull/1145))
- `Model.from_spec(..., sparse=True)` keeps a `shift` whose offset differs per entity sparse, without a `PerformanceWarning`. ([\#1145](https://github.com/PyPSA/linopy/pull/1145))
Expand Down
11 changes: 7 additions & 4 deletions doc/user-guide/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,10 +111,13 @@ data, and let linopy build it.
data in `model.parameters`, reading results, saving and rebuilding,
typesetting and variants.

linopy supports a subset of mathspec. A constraint row that absence
leaves without terms (a `missing: absent` parameter, a `missing: 0` parameter such as `0 == load`, or a `where:`-masked
variable inside a sum) is refused, where mathspec skips and reports it;
linopy is stricter here ([\#993](https://github.com/PyPSA/linopy/issues/993)). `store=True` stores
linopy supports a subset of mathspec. A constraint row that the data leaves
without a variable term (a `missing: 0` parameter such as `0 == load`, a shift
past its edge, or a sum whose terms are all absent or `where:`-masked) reads
`constant sense constant`. linopy drops it with a `linopy.EmptiedRowWarning`
where it holds and refuses it where it cannot, e.g. `0 == 5`. mathspec skips
and reports such a row; linopy is stricter here
([\#993](https://github.com/PyPSA/linopy/issues/993)). `store=True` stores
only relations of one key column and one value column, and raises
`SpecDataError` for any other.

Expand Down
14 changes: 9 additions & 5 deletions doc/user-guide/spec-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,11 +251,15 @@ variables:
propagate to the whole row. An `absent` row read outside a sum takes the
row with it, so `x <= c` has no row where `c` is absent; write
`where: c` on the constraint to make that visible in the spec.
- Inside a sum, an absent term is left out. A row the data leaves without any
variable term, such as a sum over variables a `where:` masked away
entirely, is refused rather than dropped: nothing is left to constrain, and
the row would leave the problem without saying so. Mask it with a `where:`
on the constraint ([\#993](https://github.com/PyPSA/linopy/issues/993)).
- Inside a sum, an absent term is left out, and a sum of absent terms only
is 0. A row the data leaves without any variable term, such as a sum over
variables a `where:` masked away entirely, a shift past its edge or a
coefficient of 0, reads `constant sense constant`, compared within a
tolerance of `1e-9`. Where it holds, e.g. `0 <= 5`, the row is dropped with
a `linopy.EmptiedRowWarning`. Where it cannot, e.g. `0 >= 5`, it is refused
with the first such row: dropping it would hide an infeasible problem.
Mask such rows with a `where:` on the constraint
([\#993](https://github.com/PyPSA/linopy/issues/993)).
- linopy carries a product of two variables in the objective only. A
constraint whose spec multiplies two variables builds wherever the data
leaves no such product on a row its `where:` keeps: its coefficient is
Expand Down
2 changes: 2 additions & 0 deletions linopy/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
GREATER_EQUAL,
LESS_EQUAL,
AbsorbedTermsWarning,
EmptiedRowWarning,
EvolvingAPIWarning,
PerformanceWarning,
)
Expand All @@ -45,6 +46,7 @@
"Constraints",
"Constraint",
"EQUAL",
"EmptiedRowWarning",
"PerformanceWarning",
"EvolvingAPIWarning",
"GREATER_EQUAL",
Expand Down
14 changes: 14 additions & 0 deletions linopy/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,20 @@ class AbsorbedTermsWarning(UserWarning):
SOS_BIG_M_ATTR = "big_m_upper"


class EmptiedRowWarning(UserWarning):
"""
Signals constraint rows that ``Model.from_spec`` dropped because the data
left them without a variable term and they hold, e.g. ``0 <= 5``.

Filter it with::

import warnings
import linopy

warnings.filterwarnings("ignore", category=linopy.EmptiedRowWarning)
"""


class EvolvingAPIWarning(FutureWarning):
"""
Signals a newly-added API whose details may evolve in minor releases.
Expand Down
165 changes: 111 additions & 54 deletions linopy/spec/builder.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,17 @@

import operator
import warnings
from collections.abc import Callable
from dataclasses import replace
from typing import cast

import numpy as np
import xarray as xr
from mathspec import program as ms

from linopy.constants import FACTOR_DIM, TERM_DIM
from linopy.constants import FACTOR_DIM, TERM_DIM, EmptiedRowWarning
from linopy.expressions import LinearExpression, QuadraticExpression
from linopy.restrict.matrix import DROPPED_ROW_TOL
from linopy.spec.context import Context, Term, Value, as_expression, is_empty
from linopy.spec.errors import SpecDataError, first_coordinates
from linopy.spec.evaluator import (
Expand All @@ -36,6 +38,11 @@
from linopy.variables import Variable

_FLIPPED: dict[str, ms.ConstraintSense] = {"==": "==", "<=": ">=", ">=": "<="}
_VIOLATION: dict[str, Callable[[xr.DataArray], xr.DataArray]] = {
"==": abs,
"<=": lambda gap: gap,
">=": operator.neg,
}
_SENSE = {"minimize": "min", "maximize": "max"}


Expand Down Expand Up @@ -139,7 +146,6 @@ def _constraints(ctx: Context) -> None:
ctx.memo.skipped(row.lhs, row.rhs)
continue
rows = evaluate_where(row.where, ctx)
mask = as_linopy_mask(rows)
scoped = replace(ctx, rows=rows, subject=f"constraint '{name}'")
refuse_holes(scoped.subject, [row.lhs, row.rhs], ctx, rows)
lhs, rhs, sense = row.lhs, row.rhs, row.sense
Expand All @@ -151,9 +157,11 @@ def _constraints(ctx: Context) -> None:
term = cast(Term, term)
if any(term.sizes[d] == 0 for d in term.coord_dims):
continue
_check_live(name, row, term, other, rows, ctx)
sides = [term, other] if isinstance(other, Term) else [term]
if all(as_expression(side).is_constant for side in sides):
emptied = _emptied(term, other, rows)
if bool(emptied.any()):
_check_emptied(name, row, term, other, emptied)
rows = rows & ~emptied
if _constant(term, other):
continue
if isinstance(other, xr.DataArray):
const = constant_of(term)
Expand All @@ -162,10 +170,21 @@ def _constraints(ctx: Context) -> None:
)
term, other = carried(term, other)
ctx.model.add_constraints(
_framed(term, row.dims), sense, other, name=ctx.rename(name), mask=mask
_framed(term, row.dims),
sense,
other,
name=ctx.rename(name),
mask=as_linopy_mask(rows),
)


def _constant(*sides: Value) -> bool:
"""Whether none of *sides*, the two of a comparison or an objective, holds a variable term."""
return all(
not isinstance(side, Term) or as_expression(side).is_constant for side in sides
)


def _linear(name: str, side: Value, rows: xr.DataArray) -> Value:
"""
One *side* of a constraint as a linear expression, refused where the data leaves
Expand All @@ -190,66 +209,89 @@ def _linear(name: str, side: Value, rows: xr.DataArray) -> Value:
return LinearExpression(side.data.assign(vars=vars), side.model)


def _check_live(
def _emptied(term: Term, other: Value, rows: xr.DataArray) -> xr.DataArray:
"""
The rows *rows* keeps where neither side holds a variable term and neither is absent.

An absent row is not emptied: absence spread through arithmetic, a masked
variable or a vacated translation, takes the row with it. A row of a bare
variable exists exactly where the variable does, so it is never emptied.
"""
held = _held(term) | _held(other)
absent = _constant_of(term).isnull() | _constant_of(other).isnull()
return rows & ~held & ~absent


def _held(side: Value) -> xr.DataArray:
"""Per row, whether *side* holds a variable term."""
if isinstance(side, Variable):
return side.labels != -1
if not isinstance(side, LinearExpression):
return xr.DataArray(False)
store = side._store
return store.dataarray(store.has_terms(), "held")


def _constant_of(side: Value) -> xr.DataArray:
"""The constant of *side*, NaN where it is absent."""
if isinstance(side, Variable):
return xr.where(side.labels != -1, 0.0, np.nan)
return constant_of(side)


def _check_emptied(
name: str,
declared: ms.Constraint,
term: Term,
other: Value,
rows: xr.DataArray,
ctx: Context,
emptied: xr.DataArray,
) -> None:
"""
Refuse a row the data emptied of every variable term.

Such a row reads as ``0 sense rhs``: linopy carries no column there, the
row leaves the problem and the constraint silently stops binding. Where
every variable of the row declares a ``missing:`` number, the constant
left is what the math says, so a row whose two sides are equal there is
warned about rather than refused.
An absent row is not emptied: absence spread through arithmetic, a masked
variable or a vacated translation, takes the row with it. A bare variable
term is never emptied: a row of it exists exactly where the variable does.
Drop with a warning the rows *emptied* names where they hold, refuse them where they cannot.

The data left such a row without a variable term, so it reads as
``constant sense constant``, compared as written within
``DROPPED_ROW_TOL``, the tolerance of :meth:`linopy.Model.restrict`. A
row that holds constrains nothing; one that cannot hold makes the
problem infeasible, and linopy, which carries no row without a term,
would drop it silently.
"""
if isinstance(term, Variable):
return
store = cast(LinearExpression, term)._store
alive = store.dataarray(store.has_terms() | np.isnan(store.const), "alive")
dead = rows & ~alive
if not bool(dead.any()):
return
said = (
f"constraint '{name}': {int(dead.sum())} row(s) hold no variable term once the data "
f"is attached, the first at {first_coordinates(dead, 3)}. Nothing is left to constrain there, so "
f"the row leaves the problem without saying so."
)
valued = all(
ctx.program.variables[v].missing != "absent"
for v in ms.variables_of(declared.lhs, declared.rhs)
)
if valued and not _binds(other, constant_of(term), dead):
warnings.warn(
f"{said} Every variable there takes its missing: number and the two sides "
f"are equal, so the row is trivially true.",
UserWarning,
stacklevel=2,
lhs, rhs = _constant_of(term), _constant_of(other)
if flipped(declared):
lhs, rhs = rhs, lhs
failing = emptied & (_VIOLATION[declared.sense](lhs - rhs) > DROPPED_ROW_TOL)
said = f"constraint '{name}': {{}} row(s) hold no variable term once the data is attached"
if bool(failing.any()):
raise SpecDataError(
f"{said.format(int(failing.sum()))} and cannot hold, the first at "
f"{_quoted(failing, lhs, rhs, declared.sense)}.\n"
f" Mask them out with a where on the constraint, if the row should not exist there.\n"
f" Fix the data that empties the row, if the row is meant to bind."
)
return
supply = (
" Supply the rows of the variables, if the row is meant to bind."
if valued
else " Declare missing: 0 on the variables, if an absent term is a zero there."
warnings.warn(
f"{said.format(int(emptied.sum()))}, the first at "
f"{_quoted(emptied, lhs, rhs, declared.sense)}. Each holds, so it is dropped.",
EmptiedRowWarning,
stacklevel=2,
)
raise SpecDataError(
f"{said}\n"
f" Mask them out with a where on the constraint, if the row should not exist there.\n"
f"{supply}"


def _quoted(
flags: xr.DataArray, lhs: xr.DataArray, rhs: xr.DataArray, sense: str
) -> str:
"""The first row *flags* names, with the constants its two sides reduce to."""
first = np.unravel_index(int(np.argmax(flags.values)), flags.shape)
at = dict(zip(flags.dims, first))
left, right = (
float(side.broadcast_like(flags).transpose(*flags.dims)[at])
for side in (lhs, rhs)
)
return f"{first_coordinates(flags, 1)}: {_number(left)} {sense} {_number(right)}"


def _binds(other: Value, const: xr.DataArray, dead: xr.DataArray) -> bool:
"""Whether the side without the variable term differs from the constant *const* left on a row *dead* names."""
differs = xr.DataArray(other) != const
return bool((differs & dead).any())
def _number(value: float) -> str:
"""*value* in its shortest exact form, an integral one without a fraction."""
return repr(value).removesuffix(".0")


def _objective(ctx: Context) -> None:
Expand All @@ -258,6 +300,10 @@ def _objective(ctx: Context) -> None:
return
refuse_holes("the objective", [declared.expression], ctx)
expr = evaluate(declared.expression, ctx)
if _constant(expr):
_forget_terms(ctx)
expr = evaluate(declared.expression, replace(ctx, zeroes=False))
_forget_terms(ctx)
if not isinstance(expr, Term):
raise SpecDataError(
"the objective carries no variable term once the data is attached, so there is nothing to optimize"
Expand All @@ -267,6 +313,17 @@ def _objective(ctx: Context) -> None:
)


def _forget_terms(ctx: Context) -> None:
"""Drop the terms the memo of *ctx* keeps, so the next evaluation makes them again."""
values = ctx.memo.values
for key in [
k
for k, v in values.items()
if isinstance(v, LinearExpression | QuadraticExpression)
]:
del values[key]


def _linear_if_no_product(expr: Term) -> Term:
"""
A quadratic *expr* whose products the data zeroes, as the linear expression it is.
Expand Down
3 changes: 3 additions & 0 deletions linopy/spec/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,8 @@ class Context:
``subject`` names what is being built in its refusals. ``memo`` is
shared by every context replaced from one. ``rename`` gives the model
name of a variable or constraint the program declares.
``zeroes`` makes a term the data scales by zero the constant it is
there, so a row left with such terms alone holds no variable term.
"""

model: Model
Expand All @@ -150,6 +152,7 @@ class Context:
subject: str = "the spec"
memo: Memo = field(default_factory=Memo, compare=False, repr=False)
rename: Callable[[str], str] = field(default=str, compare=False, repr=False)
zeroes: bool = True


def is_empty(dims: Iterable[str], ctx: Context) -> bool:
Expand Down
Loading
Loading