Skip to content

USE 530 - Updated to TDA v5 and python projects spec alignment #50

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ghukill merged 2 commits into from
May 1, 2026
Merged

USE 530 - Updated to TDA v5 and python projects spec alignment #50

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9093daf
Update TDA to version 5
ghukill May 1, 2026
6f1a1b6
Gentle Python project spec alignment
ghukill May 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 13 additions & 12 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,29 +1,30 @@
default_language_version:
python: python3.12

default_stages:
- pre-push

repos:
- repo: local
hooks:
- id: black-apply
name: black-apply
entry: uv run black
- id: ruff-format
name: ruff-format
entry: uv run ruff format --diff
language: system
pass_filenames: true
types: ["python"]

- id: mypy
name: mypy
entry: uv run mypy
language: system
pass_filenames: true
types: ["python"]
exclude: "tests/"
- id: ruff-apply
name: ruff-apply
entry: uv run ruff check --fix
exclude: "(tests/|output/|migrations/)"

- id: ruff-check
name: ruff-check
entry: uv run ruff check
language: system
pass_filenames: true
types: ["python"]
- id: pip-audit
name: pip-audit
entry: uv run pip-audit --ignore-vuln CVE-2025-2953 --ignore-vuln CVE-2025-3730
language: system
pass_filenames: false
42 changes: 24 additions & 18 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,22 +13,26 @@ help: # Preview Makefile commands
/^[-_[:alpha:]]+:.?*#/ { printf " %-15s%s\n", $$1, $$2 }' $(MAKEFILE_LIST)

# ensure OS binaries aren't called if naming conflict with Make recipes
.PHONY: help venv install update test coveralls lint black mypy ruff safety lint-apply black-apply ruff-apply check-arch dist-dev publish-dev docker-clean
.PHONY: help venv install update test coveralls lint ruff-format mypy ruff lint-fix security validate-arch ensure-builder dist-dev-gpu dist-dev-cpu dist-dev-all publish-dev-gpu publish-dev-cpu publish-dev-all docker-clean

##############################################
# Python Environment and Dependency commands
##############################################

install: .venv .git/hooks/pre-commit # Install Python dependencies and create virtual environment if not exists
install: .venv .git/hooks/pre-commit .git/hooks/pre-push # Install Python dependencies and create virtual environment if not exists
uv sync --group dev --group local

.venv: # Creates virtual environment if not found
@echo "Creating virtual environment at .venv..."
uv venv .venv

.git/hooks/pre-commit: # Sets up pre-commit hook if not setup
@echo "Installing pre-commit hooks..."
uv run pre-commit install
.git/hooks/pre-commit: # Sets up pre-commit commit hooks if not setup
@echo "Installing pre-commit commit hooks..."
uv run pre-commit install --hook-type pre-commit

.git/hooks/pre-push: # Sets up pre-commit push hooks if not setup
@echo "Installing pre-commit push hooks..."
uv run pre-commit install --hook-type pre-push

venv: .venv # Create the Python virtual environment

Expand All @@ -48,31 +52,33 @@ coveralls: test # Write coverage data to an LCOV report
uv run coverage lcov -o ./coverage/lcov.info

####################################
# Code quality and safety commands
# Code linting, formatting, and security
####################################

lint: black mypy ruff safety # Run linters
lint: ruff-format mypy ruff # Run linting, alerts only, no code changes

black: # Run 'black' linter and print a preview of suggested changes
uv run black --check --diff .
ruff-format: # Run 'ruff format' and print a preview of suggested changes
uv run ruff format --diff .

mypy: # Run 'mypy' linter
uv run mypy .

ruff: # Run 'ruff' linter and print a preview of errors
uv run ruff check .

safety: # Check for security vulnerabilities
uv run pip-audit --ignore-vuln CVE-2025-2953 --ignore-vuln CVE-2025-3730

lint-apply: black-apply ruff-apply # Apply changes with 'black' and resolve 'fixable errors' with 'ruff'

black-apply: # Apply changes with 'black'
uv run black .

ruff-apply: # Resolve 'fixable errors' with 'ruff'
lint-fix: # Run linting, auto fix behaviors where supported
uv run ruff format .
uv run ruff check --fix .

# CVE-2025-2953 = torch 2.6; acceptable risk, keeping until we move away from torch 2.6
# CVE-2025-3730 = torch 2.6; acceptable risk, keeping until we move away from torch 2.6
# CVE-2026-1839 = transformers 4.57; acceptable risk, keeping until we move to transformers 5.x
Comment on lines +73 to +75

@ghukill ghukill May 1, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a bit of an experiment. These CVE ignores are likely somewhat long living, which we can be okay with. These are not risks to us, but we can't upgrade torch or transformers easily based on the model we are using.

Thankfully, pip-audit does not run on git commit or push, or Github CI, so we won't encounter these unless running make security. That said, I wanted a little context as to why the dependencies are pinned. I'm realizing we could put these notes in those dependency pins... but this feels okay for now.

security: # Check for security vulnerabilities
uv run pip-audit \
--ignore-vuln CVE-2025-2953 \
--ignore-vuln CVE-2025-3730 \
--ignore-vuln CVE-2026-1839


####################################
# Developer Build and Deploy Commands for Dev environment in AWS
Expand Down
4 changes: 3 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ A CLI application for creating embeddings for TIMDEX.
- To update dependencies: `make update`
- To run unit tests: `make test`
- To lint the repo: `make lint`
- To run the app: `embeddings --help`
- To apply formatting and fixable lint errors: `make lint-fix`
- To run security checks: `make security`
- To run the app: `uv run embeddings --help`
- see below for more details about running the CLI with `.env` files and arguments

### Building Docker Images
Expand Down
4 changes: 2 additions & 2 deletions embeddings/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,7 @@ def create_embeddings(

# get ETL run records
if run_id:
timdex_records = timdex_dataset.read_dicts_iter(
timdex_records = timdex_dataset.records.read_dicts_iter(
table="records",
columns=[
"timdex_record_id",
Expand All @@ -297,7 +297,7 @@ def create_embeddings(

# get current records for a source
else:
timdex_records = timdex_dataset.read_dicts_iter(
timdex_records = timdex_dataset.records.read_dicts_iter(
table="current_records",
columns=[
"timdex_record_id",
Expand Down
1 change: 0 additions & 1 deletion embeddings/strategies/processor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ def create_embedding_inputs(

# loop through records and apply all strategies, yielding an EmbeddingInput for each
for timdex_dataset_record in timdex_dataset_records:

# decode and parse the TIMDEX JSON record once for all requested strategies
transformed_record_raw = timdex_dataset_record["transformed_record"]
timdex_record = json.loads(
Expand Down
4 changes: 0 additions & 4 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ dependencies = [

[dependency-groups]
dev = [
"black>=25.1.0",
"coveralls>=4.0.1",
"ipython>=9.6.0",
"mypy>=1.17.1",
Expand All @@ -41,9 +40,6 @@ dlc_arm64_cpu = []
# dependency group for amd64 / GPU AWS Deep Learning Container (DLC) base image
dlc_amd64_gpu = []

[tool.black]
line-length = 90

[tool.mypy]
disallow_untyped_calls = true
disallow_untyped_defs = true
Expand Down
5 changes: 2 additions & 3 deletions tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@

import pytest
from click.testing import CliRunner
from timdex_dataset_api import TIMDEXDataset
from timdex_dataset_api.record import DatasetRecord
from timdex_dataset_api import DatasetRecord, TIMDEXDataset

from embeddings.embedding import Embedding, EmbeddingInput
from embeddings.models import registry
Expand Down Expand Up @@ -160,7 +159,7 @@ def dataset_with_records(tmp_path) -> TIMDEXDataset:
)

timdex_dataset = TIMDEXDataset(str(dataset_path))
timdex_dataset.write(records)
timdex_dataset.records.write(records)
timdex_dataset.metadata.rebuild_dataset_metadata()

# reload and return dataset
Expand Down
2 changes: 1 addition & 1 deletion tests/test_cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ def test_create_embeddings_writes_to_timdex_dataset(
timdex_dataset = TIMDEXDataset(location=dataset_with_records.location)

# assert embeddings written
assert Path(timdex_dataset.embeddings.data_embeddings_root).exists()
assert Path(timdex_dataset.embeddings.data_root).exists()
embeddings_df = timdex_dataset.embeddings.read_dataframe(run_id="run-1")
assert len(embeddings_df) == 2

Expand Down
Loading
Loading