Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions doc/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ linopy 1.0 makes the strict arithmetic convention the only behaviour and puts de
- A sparse constraint keeps `rhs`, `sign`, `scaling` and `dual` on its full grid like a dense one; masked rows have label `-1`. ([\#1047](https://github.com/PyPSA/linopy/issues/1047))
- `Constraint.update` raises when a new `lhs` or `rhs` adds dimensions, and `update(variables=...)` accepts only a `Variable`. After `update(coeffs=...)` or `update(variables=...)`, `coeffs` and `vars` keep the `_term` dimension. `sanitize_zeros` removes near-zero terms, so `_term` can shrink. `set_blocks` stores the `blocks` field. ([\#1046](https://github.com/PyPSA/linopy/issues/1046), [\#1047](https://github.com/PyPSA/linopy/issues/1047))
- `Constraint.active_labels`, `to_matrix` and `to_polars` raise on a constraint without labels.
- A constraint row without a nonzero term stays in the model and reaches the solver, e.g. a sum whose terms are all absent: `x.where(mask).sum("j") >= 5` is `0 >= 5` where the mask is all `False`, so the model is infeasible where 0.x dropped the row. `ncons`, the matrices and the duals count such rows. Mask them, e.g. `add_constraints(lhs >= 5, mask=lhs.has_terms)`, or fix the data. ([\#1170](https://github.com/PyPSA/linopy/issues/1170), [\#1183](https://github.com/PyPSA/linopy/pull/1183))

*Saved models*

Expand Down Expand Up @@ -153,7 +154,9 @@ linopy 1.0 makes the strict arithmetic convention the only behaviour and puts de

*Solvers*

- A constraint row without a nonzero term, e.g. `0 * x >= 5`, reaches the solver instead of being dropped; `solve` and `to_file` warn when it cannot hold. ([\#1170](https://github.com/PyPSA/linopy/issues/1170), [\#1183](https://github.com/PyPSA/linopy/pull/1183))
- With `add_objective(..., scaling=...)`, the solve log reports objective and dual bound in model units. ([\#1077](https://github.com/PyPSA/linopy/issues/1077))
- GLPK: an infeasible MIP, or an infeasible LP solved without presolve, is reported as `infeasible` instead of `ok`. ([\#1183](https://github.com/PyPSA/linopy/pull/1183))
- CPLEX: "optimal with unscaled infeasibilities" is reported as `imprecise` and its solution is imported. ([\#368](https://github.com/PyPSA/linopy/issues/368))
- HiGHS: a solve that stops at a limit without a solution no longer returns all-zero values, and a MIP no longer returns all-zero duals. `log_fn` and solver options also apply to a re-solve of a direct-API model. ([\#1130](https://github.com/PyPSA/linopy/issues/1130))
- The `solvers` extra excludes highspy 1.16.0. Its MPS writer drops columns that appear only in the quadratic objective, so Gurobi and COPT reject the file. ([\#1149](https://github.com/PyPSA/linopy/pull/1149))
Expand Down
8 changes: 8 additions & 0 deletions doc/user-guide/design/convention.md
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,14 @@ Each slot becomes one constraint row. An absent slot yields no row — absence
propagated into a comparison drops the constraint there, the same outcome as
masking it.

A present slot always yields a row, also one without a nonzero term, e.g. a sum
over absent terms (§13) or `0 * x`. `0 * x >= 5` reaches the solver as `0 >= 5`,
so the model is infeasible; `solve` and `to_file` name such rows in a warning.
Only explicit steps drop a present row: `sanitize_missings()`,
`sanitize_infinities()` (run by `solve`; it drops `<= inf` and `>= -inf`
rows) and [`restrict()`][linopy.model.Model.restrict], which drops a row that keeps no term
in the window if it holds within `tol`.

### §13. Reductions skip absent slots

Reductions collapse a dimension rather than combining two operands, so the
Expand Down
1 change: 1 addition & 0 deletions doc/user-guide/migrating-to-v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ Every row is a legacy guess that is now an explicit rule.
| A reindexing `join=` that **creates labels** neither operand had | The join still returns the union of the coordinates; the created rows are filled per side — the linopy operand contributes the zero expression, the constant operand `fill_value=` (0 by default). A missing divisor now zeroes that row instead of leaving the term unscaled, and a missing numerator no longer comes out as `1 / divisor`. | Pass `fill_value=` on `.add` / `.sub` / `.mul` / `.div` to choose the value, e.g. `.div(other, join="outer", fill_value=1)` to keep the term unscaled, or `fill_value=linopy.ABSENT` to keep the created rows absent so the constraint drops there. |
| An **unlabeled** operand (numpy array, list, polars `Series`) | Pairs with the linopy operand's dimensions by *size*; an ambiguous match (a square array, or two dimensions of equal length) or no size match raises rather than guessing. | Wrap it in a `DataArray` / `Series` / `DataFrame` with named dimensions so it aligns by label. |
| A **masked / absent** variable in arithmetic | Absence propagates (the slot stays absent) instead of counting as `0`. | Decide the intent: `.fillna(0)` to keep the old "treat as zero", or leave it to propagate and drop the term. |
| A constraint row **without a nonzero term**, e.g. a `sum` over terms that are all absent | Stays in the model as `0 >= rhs` and reaches the solver, where 0.x dropped it. A row that cannot hold makes the model infeasible; `solve` and `to_file` name it in a warning. | Mask the row, e.g. `add_constraints(lhs >= rhs, mask=lhs.has_terms)`, or fix the data. |
| **Conflicting auxiliary coordinates** on a shared dim — including a leftover *scalar* coord from positional indexing (`x.isel(time=0)` keeps `time` as a scalar coord, so a cyclic/boundary constraint like `x.isel(time=0) == x.isel(time=-1)` sees `time` = first vs last) | Raises instead of silently dropping one. | `.drop_vars(name)` / `.assign_coords(name=...)`; for a leftover index coord, drop it at the source with `.isel(..., drop=True)` / `.sel(..., drop=True)`. |
| A first-class `pd.MultiIndex` **dimension** — and a per-*level* input onto it (e.g. per-`period` bounds onto a `(period, timestep)` `snapshot`) | Rejected; linopy uses a flat dimension with the levels as aux coords, and a per-level input must be mapped onto the flat dimension explicitly (no implicit projection). Affects PyPSA multi-investment models. Netcdf files written with such a dimension by linopy \< 1.0 are refused by [`read_netcdf()`][linopy.io.read_netcdf]; rebuild the model and save it again (see [Saved models (netcdf)](#saved-models-netcdf)). | `.reset_index(dim)` to flatten, then project the per-level input by its level aux coord. |
| A multi-key `groupby(...).sum()` result | A flat `group` dimension with the keys as aux coords (not a stacked `group` MultiIndex). | Select on the key aux coords; convert an existing result with `.reset_index("group")`. |
Expand Down
5 changes: 3 additions & 2 deletions doc/user-guide/restrict.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,9 @@ $$
where $x_W$ are the window's variables, $x_B$ the variables
outside it, and $x_B$ on the right-hand side the values you pass in
`fix`. In words, each outside term adds `coefficient * fixed value` to the
right-hand side. Rows that lose every term are checked and dropped when they
are satisfied.
right-hand side. Rows that have no term in the window, also those without a
term in the model, are checked against `tol` and dropped when they are
satisfied.

Without `fix`, a window that cuts a coupling raises
[`RestrictionError`][linopy.restrict.RestrictionError] naming the constraints. A window
Expand Down
23 changes: 18 additions & 5 deletions linopy/constraints.py
Original file line number Diff line number Diff line change
Expand Up @@ -578,7 +578,7 @@ def nterm(self) -> int:

@property
def ncons(self) -> int:
"""Get the number of active constraints (non-masked, with at least one valid variable)."""
"""Get the number of active constraints: rows with a label, also those without a term."""
return int(self.active_row_mask().sum())

@property
Expand Down Expand Up @@ -1004,8 +1004,8 @@ def to_matrix(
return csr, self._label_values()[active]

def active_row_mask(self) -> np.ndarray:
"""Boolean mask over raveled rows: label set and at least one variable present."""
return (self._label_values() != -1) & self._lhs._store.has_terms()
"""Boolean mask over raveled rows: label set, with or without a term."""
return self._label_values() != -1

def active_labels(self) -> np.ndarray:
"""Active constraint labels in build order, without building the CSR."""
Expand Down Expand Up @@ -1264,9 +1264,22 @@ def to_polars(self) -> pl.DataFrame:
-------
df : polars.DataFrame
"""
return self._to_polars(None)

def _to_polars(self, empty_var: int | None) -> pl.DataFrame:
"""
Long frame of :meth:`to_polars`. With ``empty_var``, a row without a
term gets the single term ``0 * empty_var``, so a writer keeps it.
"""
labels_flat = self._label_values()
mask = labels_flat != -1
counts, vars_, coeffs = self._lhs._store.terms(active=mask)
empty = mask & (counts == 0)
if empty_var is not None and empty.any():
starts = (np.cumsum(counts) - counts)[empty]
vars_ = np.insert(vars_, starts, empty_var)
coeffs = np.insert(coeffs, starts, 0.0)
counts = np.where(empty, 1, counts)
long = pl.DataFrame(
{"labels": np.repeat(labels_flat, counts), "coeffs": coeffs, "vars": vars_}
)
Expand Down Expand Up @@ -1463,8 +1476,8 @@ def ncons(self) -> int:
"""
Get the number all constraints effectively used by the model.

This excludes constraints with missing labels or where all variables
are masked (vars == -1).
This excludes rows with missing labels (-1). A row with a label but
without a term counts.
"""
return sum(con.ncons for con in self.data.values() if not con.is_indicator)

Expand Down
10 changes: 9 additions & 1 deletion linopy/io.py
Original file line number Diff line number Diff line change
Expand Up @@ -645,6 +645,13 @@ def constraints_to_file(
m, explicit_coordinate_names=explicit_coordinate_names
)
scaled = variable_scaling is not None or constraint_scaling is not None
vlabels = m.variables.label_index.vlabels
if not vlabels.size and m.constraints.ncons:
raise ValueError(
"The model has constraint rows without a term, but no variable "
"to write them with."
)
empty_var = int(vlabels[0]) if vlabels.size else None

f.write(b"\n\ns.t.\n\n")
names = list(regular)
Expand All @@ -657,7 +664,8 @@ def constraints_to_file(

for name in names:
con = regular[name]
for df in map(Constraint.to_polars, con.iterate_slices(slice_size)):
for part in con.iterate_slices(slice_size):
df = part._to_polars(empty_var)
if df.height == 0:
continue
if scaled:
Expand Down
68 changes: 64 additions & 4 deletions linopy/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
from xarray import DataArray, Dataset
from xarray.core.types import T_Chunks

from linopy import solvers
from linopy import io, solvers
from linopy.alignment import (
_as_index,
_coords_to_dict,
Expand All @@ -39,13 +39,16 @@
assigned_labels,
best_int,
coords_reorder_set,
format_coord,
maybe_replace_sign,
maybe_replace_signs,
reassign_coords,
replace_by_map,
to_path,
)
from linopy.config import options
from linopy.constants import (
EQUAL,
FACTOR_DIM,
GREATER_EQUAL,
LESS_EQUAL,
Expand Down Expand Up @@ -79,7 +82,6 @@
to_block_files,
to_cuopt,
to_cupdlpx,
to_file,
to_gurobipy,
to_highspy,
to_mosek,
Expand All @@ -94,7 +96,7 @@
from linopy.remote import RemoteHandler
from linopy.restrict.matrix import DROPPED_ROW_TOL
from linopy.scaling import scalar_scaling, validate_scaling
from linopy.semantics import check_user_nan, enforce_no_multiindex
from linopy.semantics import check_user_nan, enforce_no_multiindex, warn_outside_linopy
from linopy.solver_capabilities import solver_supports
from linopy.solvers import (
IO_APIS,
Expand Down Expand Up @@ -2222,6 +2224,7 @@ def solve(
"`m.add_objective(...)` first (e.g. `m.add_objective(0 * x)` "
"for a pure feasibility problem)."
)
self._warn_empty_rows_that_cannot_hold()
solved = remote.solve_on_remote(
self,
solver_name=solver_name,
Expand Down Expand Up @@ -2288,6 +2291,7 @@ def solve(
self.constraints.sanitize_zeros()
if sanitize_infinities:
self.constraints.sanitize_infinities()
self._warn_empty_rows_that_cannot_hold()

try:
self.solver = None # closes any previous solver
Expand Down Expand Up @@ -2636,6 +2640,7 @@ def _mock_solve(

if sanitize_infinities:
self.constraints.sanitize_infinities()
self._warn_empty_rows_that_cannot_hold()

self.objective._value = 0.0
self.status = "ok"
Expand Down Expand Up @@ -2704,7 +2709,62 @@ def reset_solution(self) -> None:

to_netcdf = to_netcdf

to_file = to_file
def to_file(
self,
fn: Path | str | None,
io_api: str | None = None,
integer_label: str = "general",
slice_size: int = 2_000_000,
progress: bool | None = None,
explicit_coordinate_names: bool = False,
) -> Path:
"""
Write out the model to an LP or MPS file, see :func:`linopy.io.to_file`.

Warns about constraint rows without a nonzero term that cannot hold.
"""
self._warn_empty_rows_that_cannot_hold()
return io.to_file(
self,
fn,
io_api=io_api,
integer_label=integer_label,
slice_size=slice_size,
progress=progress,
explicit_coordinate_names=explicit_coordinate_names,
)

def _warn_empty_rows_that_cannot_hold(self) -> None:
"""Warn about rows without a nonzero term whose ``0 sign rhs`` fails."""
found = []
for con in self.constraints.data.values():
if con.is_indicator:
continue
labels = con._label_values()
rhs = con.rhs.values.ravel()
sign = con.sign.values.ravel()
excess = np.where(sign == LESS_EQUAL, -rhs, rhs)
excess = np.where(sign == EQUAL, np.abs(rhs), excess)
rows = (labels != -1) & (excess > DROPPED_ROW_TOL)
if rows.any():
rows &= con._lhs._store.terms(active=rows)[0] == 0
found.append(labels[rows])
labels = np.concatenate([np.array([], dtype=int), *found])
if not labels.size:
return
shown = options["display_max_rows"]
names = [
f"{name}{format_coord(coord)}"
for name, coord in self.constraints.get_label_position(labels[:shown])
]
more = ", ..." if labels.size > shown else ""
warn_outside_linopy(
f"{labels.size} constraint row(s) have no nonzero term and cannot "
f"hold: {', '.join(names)}{more}. They reach the solver, which will "
"likely report the model infeasible. Mask them, or fix their "
"coefficients or right-hand side.",
UserWarning,
)

to_gurobipy = to_gurobipy

Expand Down
6 changes: 3 additions & 3 deletions linopy/restrict/matrix.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,9 +162,9 @@ def _check_dropped(
)
if violated.any():
raise RestrictionError(
"Cannot window the matrices: folding the dangling terms makes "
f"constraint labels {clabels[violated].tolist()} unsatisfiable "
f"(folded rhs {rhs[violated].tolist()} against sign "
"Cannot window the matrices: constraint labels "
f"{clabels[violated].tolist()} have no term in the window and are "
f"unsatisfiable (folded rhs {rhs[violated].tolist()} against sign "
f"{sense[violated].tolist()}). Raise `tol` or exclude the rows "
"via `drop`."
)
Expand Down
5 changes: 4 additions & 1 deletion linopy/solvers.py
Original file line number Diff line number Diff line change
Expand Up @@ -1068,7 +1068,8 @@ def _build_file(self, **build_kwargs: Any) -> None:
"lp files, disabling it."
)
explicit_coordinate_names = False
problem_fn = model.to_file(
problem_fn = linopy.io.to_file(
model,
Path(problem_fn) if not isinstance(problem_fn, Path) else problem_fn,
io_api=io_api,
explicit_coordinate_names=explicit_coordinate_names,
Expand Down Expand Up @@ -1725,6 +1726,8 @@ def _run_file(
"integer optimal": "optimal",
"integer undefined": "infeasible_or_unbounded",
"undefined": "infeasible_or_unbounded",
"infeasible (final)": "infeasible",
"integer empty": "infeasible",
}
sense = read_sense_from_problem_file(problem_fn)
io_api = read_io_api_from_problem_file(problem_fn)
Expand Down
Loading
Loading