-
Notifications
You must be signed in to change notification settings - Fork 0
Monthly reports #36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Monthly reports #36
Changes from all commits
Commits
Show all changes
37 commits
Select commit
Hold shift + click to select a range
33dc3c8
feat!: created monthly account reports trigger
edinstance 39c51b9
fix: lint and formatted code
edinstance 0470a0a
refactor: changed file structure
edinstance a8dc6b5
feat: added a cloudwatch schedule for the monthly trigger
edinstance a32d453
fix: added missing sqs enviroment variables to proccess transaction f…
edinstance 1672f90
refactor: changed sqs message handling so it is more reusable and upd…
edinstance 19bade3
feat: updated dynamodb data fetching helpers
edinstance b0878c9
feat!: updated monthly accounts reports trigger to include sqs for co…
edinstance 2677f7b
feat: added longer timeout for trigger function
edinstance 3de7d2f
feat!: created new lambda to process from sqs and moved shared logic …
edinstance 06a77d9
fix: updated powertools version
edinstance 85871d2
feat: updated code based on code rabbit suggestions
edinstance f41e6c6
fix: updated comparisons
edinstance 1dcad17
fix: formatted code
edinstance 98e22af
feat: added a dlq to the trigger and processing of monthly reports
edinstance ed80c5e
fix: added fixes from coderabbit
edinstance 42ee22b
refactor: changed report proccessing to have one step function per ac…
edinstance 23bc899
fix: added missing cloudwatch log groups
edinstance 34b5390
feat: created get account transactions lambda and also added it to th…
edinstance 011f5db
feat: created s3 bucket for reports
edinstance 082ee26
feat: created shared s3 client
edinstance 906cc64
fix: added balance to state machine input and scan params
edinstance 7919f59
feat: created lambda to create reports
edinstance 8b95612
feat: updated ses to include no reply and added options to send attac…
edinstance f9a5cbd
feat: created notify user lambda and added it to state machine
edinstance e787c61
fix: formatted code
edinstance 7a2293e
feat!: updated state machine to include retries and error handling wi…
edinstance 2cf9d4d
feat: updated unit tests for all new functions
edinstance e1fa0e9
feat: added option for api request to send a report to the user
edinstance 001cd27
fix: added check to make sure future dates cannot be requested
edinstance 01d04b7
fix: fixed formatting and linting
edinstance 684edef
fix: updated dev requirements
edinstance 47a7381
fix: fixed mismatched environment variable names
edinstance 12d3e38
feat: added input validation
edinstance 2c9d901
fix: added auto escape to jinja
edinstance 24d6c7c
fix: stopped writing file to lambda
edinstance 8bee5de
fix: fixed formatting and tests
edinstance File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
Empty file.
99 changes: 99 additions & 0 deletions
99
functions/accounts/get_account_transactions/get_account_transactions/app.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| import os | ||
| import json | ||
|
|
||
| from aws_lambda_powertools import Logger | ||
| from aws_lambda_powertools.event_handler import ( | ||
| APIGatewayRestResolver, | ||
| CORSConfig, | ||
| ) | ||
| from aws_lambda_powertools.event_handler.exceptions import ( | ||
| InternalServerError, | ||
| BadRequestError, | ||
| ) | ||
| from aws_lambda_powertools.utilities.typing import LambdaContext | ||
|
|
||
| from dynamodb import get_dynamodb_resource | ||
| from .exceptions import ValidationError | ||
| from .transaction_helpers import query_transactions | ||
|
|
||
| TRANSACTIONS_TABLE_NAME = os.environ.get("TRANSACTIONS_TABLE_NAME") | ||
| ENVIRONMENT_NAME = os.environ.get("ENVIRONMENT_NAME", "dev") | ||
| POWERTOOLS_LOG_LEVEL = os.environ.get("POWERTOOLS_LOG_LEVEL", "INFO").upper() | ||
| DYNAMODB_ENDPOINT = os.environ.get("DYNAMODB_ENDPOINT") | ||
| AWS_REGION = os.environ.get("AWS_REGION", "eu-west-2") | ||
|
|
||
| logger = Logger(service="GetAccountTransactions", level=POWERTOOLS_LOG_LEVEL) | ||
|
|
||
| app = APIGatewayRestResolver( | ||
| cors=CORSConfig(allow_headers=["Content-Type", "Authorization"]) | ||
| ) | ||
|
|
||
| dynamodb = get_dynamodb_resource(DYNAMODB_ENDPOINT, AWS_REGION, logger) | ||
| if TRANSACTIONS_TABLE_NAME: | ||
| table = dynamodb.Table(TRANSACTIONS_TABLE_NAME) | ||
| logger.debug(f"Initialized DynamoDB table: {TRANSACTIONS_TABLE_NAME}") | ||
| else: | ||
| logger.critical("FATAL: TRANSACTIONS_TABLE_NAME environment variable not set!") | ||
| table = None | ||
|
|
||
|
|
||
| @app.get("/accounts/<account_id>/transactions") | ||
| def get_account_transactions(account_id: str): | ||
| try: | ||
| period = app.current_event.get_query_string_value("period", default_value=None) | ||
| start = app.current_event.get_query_string_value("start", default_value=None) | ||
| end = app.current_event.get_query_string_value("end", default_value=None) | ||
|
|
||
| result = query_transactions( | ||
| table=table, | ||
| account_id=account_id, | ||
| logger=logger, | ||
| period=period, | ||
| start=start, | ||
| end=end, | ||
| ) | ||
| return result | ||
|
|
||
| except ValidationError as ve: | ||
| logger.warning(f"Validation error: {ve}") | ||
| raise BadRequestError(str(ve)) | ||
| except Exception as e: | ||
| logger.error(f"Error fetching transactions: {e}", exc_info=True) | ||
| raise InternalServerError("Internal server error") | ||
|
|
||
|
|
||
| @logger.inject_lambda_context | ||
| def lambda_handler(event, context: LambdaContext): | ||
| logger.append_keys(request_id=context.aws_request_id) | ||
| logger.info(f"Processing request in {ENVIRONMENT_NAME}") | ||
|
|
||
| if not table: | ||
| logger.error("DynamoDB table resource is not initialized") | ||
| raise InternalServerError("Server configuration error") | ||
|
|
||
| # Detect Step Functions or API Gateway | ||
| if "httpMethod" in event or "requestContext" in event: | ||
| return app.resolve(event, context) | ||
| else: | ||
| account_id = event.get("accountId") | ||
| if not account_id: | ||
| return { | ||
| "statusCode": 400, | ||
| "body": json.dumps({"error": "Missing accountId"}), | ||
| } | ||
|
|
||
| try: | ||
| result = query_transactions( | ||
| table=table, account_id=account_id, logger=logger | ||
| ) | ||
|
|
||
| response = { | ||
| **event, | ||
| "transactions": result.get("transactions", result), | ||
| } | ||
|
|
||
| return response | ||
|
|
||
| except Exception as e: | ||
| logger.error(f"Error fetching transactions: {e}", exc_info=True) | ||
| return {"statusCode": 500, "body": json.dumps({"error": str(e)})} | ||
68 changes: 68 additions & 0 deletions
68
functions/accounts/get_account_transactions/get_account_transactions/date_helpers.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| from datetime import datetime, timedelta, timezone | ||
| import calendar | ||
|
|
||
| from .exceptions import ValidationError | ||
|
|
||
|
|
||
| def get_date_range(period: str = None, start: str = None, end: str = None): | ||
| # --- Validation rules --- | ||
| if period and (start or end): | ||
| raise ValidationError("Cannot combine 'period' with 'start'/'end'") | ||
|
|
||
| if (start and not end) or (end and not start): | ||
| raise ValidationError("Both 'start' and 'end' must be provided together") | ||
|
|
||
| # --- Custom range --- | ||
| if start and end: | ||
| try: | ||
| start_dt = datetime.strptime(start, "%Y-%m-%d").replace(tzinfo=timezone.utc) | ||
| end_dt = datetime.strptime(end, "%Y-%m-%d").replace( | ||
| hour=23, minute=59, second=59, tzinfo=timezone.utc | ||
| ) | ||
| except ValueError: | ||
| raise ValidationError("Invalid date format, must be YYYY-MM-DD") | ||
|
|
||
| if end_dt < start_dt: | ||
| raise ValidationError("'end' date must be after 'start' date") | ||
|
|
||
| statement_period = ( | ||
| f"{start_dt.strftime('%Y-%m-%d')}_to_{end_dt.strftime('%Y-%m-%d')}" | ||
| ) | ||
|
|
||
| # --- Period (month) --- | ||
| elif period: | ||
| try: | ||
| year, month = map(int, period.split("-")) | ||
| start_dt = datetime(year, month, 1, tzinfo=timezone.utc) | ||
| last_day_num = calendar.monthrange(year, month)[1] | ||
| end_dt = datetime( | ||
| year, month, last_day_num, 23, 59, 59, tzinfo=timezone.utc | ||
| ) | ||
| except Exception: | ||
| raise ValidationError("Invalid period format, must be YYYY-MM") | ||
|
|
||
| statement_period = start_dt.strftime("%Y-%m") | ||
|
|
||
| # --- Default: last month --- | ||
| else: | ||
| today = datetime.now(timezone.utc) | ||
| first_day_this_month = datetime(today.year, today.month, 1, tzinfo=timezone.utc) | ||
| last_day_last_month = first_day_this_month - timedelta(days=1) | ||
| start_dt = datetime( | ||
| last_day_last_month.year, last_day_last_month.month, 1, tzinfo=timezone.utc | ||
| ) | ||
| end_dt = datetime( | ||
| last_day_last_month.year, | ||
| last_day_last_month.month, | ||
| last_day_last_month.day, | ||
| 23, | ||
| 59, | ||
| 59, | ||
| tzinfo=timezone.utc, | ||
| ) | ||
| statement_period = start_dt.strftime("%Y-%m") | ||
|
|
||
| start_iso = start_dt.strftime("%Y-%m-%dT%H:%M:%SZ") | ||
| end_iso = end_dt.strftime("%Y-%m-%dT%H:%M:%SZ") | ||
|
|
||
| return statement_period, start_iso, end_iso |
2 changes: 2 additions & 0 deletions
2
functions/accounts/get_account_transactions/get_account_transactions/exceptions.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| class ValidationError(Exception): | ||
| pass |
35 changes: 35 additions & 0 deletions
35
functions/accounts/get_account_transactions/get_account_transactions/transaction_helpers.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| from aws_lambda_powertools import Logger | ||
| from boto3.dynamodb.conditions import Key | ||
|
|
||
| from . import date_helpers | ||
|
|
||
|
|
||
| def query_transactions( | ||
| table, | ||
| account_id: str, | ||
| logger: Logger, | ||
| period: str = None, | ||
| start: str = None, | ||
| end: str = None, | ||
| descending=False, | ||
| ): | ||
|
edinstance marked this conversation as resolved.
|
||
| statement_period, start_iso, end_iso = date_helpers.get_date_range( | ||
| period, start, end | ||
| ) | ||
|
|
||
| logger.info( | ||
| f"Querying transactions for account {account_id} " | ||
| f"from {start_iso} to {end_iso} (period {statement_period})" | ||
| ) | ||
|
|
||
| response = table.query( | ||
| IndexName="AccountDateIndex", | ||
| KeyConditionExpression=Key("accountId").eq(account_id) | ||
| & Key("createdAt").between(start_iso, end_iso), | ||
| ScanIndexForward=not descending, | ||
| ) | ||
|
|
||
| return { | ||
| "statementPeriod": statement_period, | ||
| "transactions": response.get("Items", []), | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| aws_lambda_powertools==3.17.0 | ||
| boto3==1.38.13 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,2 @@ | ||
| aws_lambda_powertools==3.12.0 | ||
| aws_lambda_powertools==3.17.0 | ||
| boto3==1.38.13 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,2 @@ | ||
| aws_lambda_powertools==3.12.0 | ||
| aws_lambda_powertools==3.17.0 | ||
| boto3==1.38.13 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,2 @@ | ||
| aws_lambda_powertools==3.12.0 | ||
| aws_lambda_powertools==3.17.0 | ||
| boto3==1.38.13 |
Empty file.
Empty file.
Empty file.
Empty file.
78 changes: 78 additions & 0 deletions
78
functions/monthly_reports/accounts/create_report/create_report/app.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| import os | ||
|
|
||
| from aws_lambda_powertools import Logger | ||
| from aws_lambda_powertools.utilities.typing import LambdaContext | ||
| from botocore.exceptions import ClientError | ||
|
|
||
| from .exceptions import ReportGenerationError, ReportTemplateError, ReportUploadError | ||
| from s3 import get_s3_client | ||
| from .generate_pdf import generate_transactions_pdf | ||
|
|
||
| REPORTS_BUCKET = os.environ.get("REPORTS_BUCKET") | ||
| POWERTOOLS_LOG_LEVEL = os.environ.get("POWERTOOLS_LOG_LEVEL") | ||
| AWS_REGION = os.environ.get("AWS_REGION") | ||
|
|
||
| logger = Logger(service="CreateAccountsReport", level=POWERTOOLS_LOG_LEVEL) | ||
|
|
||
| s3 = get_s3_client(AWS_REGION, logger) | ||
|
|
||
|
|
||
| def lambda_handler(event, _context: LambdaContext): | ||
| logger.info(f"Received event: {event}") | ||
|
|
||
| try: | ||
| required = [ | ||
| "accountId", | ||
| "userId", | ||
| "statementPeriod", | ||
| "transactions", | ||
| "accountBalance", | ||
| ] | ||
| missing = [k for k in required if k not in event] | ||
|
|
||
| if missing: | ||
| logger.error(f"Missing required fields: {missing}") | ||
| raise ReportGenerationError(f"Invalid event: missing {missing}") | ||
|
|
||
| # Generate PDF | ||
| pdf_bytes = generate_transactions_pdf(event=event, logger=logger) | ||
|
|
||
| logger.info("PDF generated successfully") | ||
|
|
||
| # Store in S3 | ||
| s3_key = f"{event['accountId']}/{event['statementPeriod']}.pdf" | ||
| try: | ||
| s3.put_object( | ||
| Bucket=REPORTS_BUCKET, | ||
| Key=s3_key, | ||
| Body=pdf_bytes, | ||
| ContentType="application/pdf", | ||
| ) | ||
|
edinstance marked this conversation as resolved.
|
||
| except ClientError as e: | ||
| logger.exception("Failed to upload report to S3") | ||
| raise ReportUploadError(f"S3 upload failed: {str(e)}") from e | ||
|
|
||
| logger.info("Generating PDF uploaded to S3") | ||
|
|
||
| try: | ||
| presigned_url = s3.generate_presigned_url( | ||
| "get_object", | ||
| Params={"Bucket": REPORTS_BUCKET, "Key": s3_key}, | ||
| ExpiresIn=3600, | ||
| ) | ||
| except ClientError as e: | ||
| logger.exception("Failed to generate presigned URL") | ||
| raise ReportUploadError(f"Presigned URL generation failed: {str(e)}") from e | ||
|
|
||
| logger.info("Presigned URL generated successfully") | ||
|
|
||
| return { | ||
| "reportUrl": presigned_url, | ||
| "accountId": event["accountId"], | ||
| "userId": event["userId"], | ||
| "statementPeriod": event["statementPeriod"], | ||
| } | ||
|
|
||
| except (ReportGenerationError, ReportTemplateError, ReportUploadError): | ||
| logger.exception("Report generation failed") | ||
| raise | ||
10 changes: 10 additions & 0 deletions
10
functions/monthly_reports/accounts/create_report/create_report/exceptions.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| class ReportGenerationError(Exception): | ||
| """Raised when PDF generation fails.""" | ||
|
|
||
|
|
||
| class ReportTemplateError(Exception): | ||
| """Raised when the Jinja2 template is missing or invalid.""" | ||
|
|
||
|
|
||
| class ReportUploadError(Exception): | ||
| """Raised when uploading to S3 fails.""" |
44 changes: 44 additions & 0 deletions
44
functions/monthly_reports/accounts/create_report/create_report/generate_pdf.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| import io | ||
| import os | ||
| from datetime import datetime, timezone | ||
|
|
||
| from aws_lambda_powertools import Logger | ||
| from jinja2 import Environment, FileSystemLoader, TemplateNotFound, select_autoescape | ||
| from xhtml2pdf import pisa | ||
|
|
||
| from .exceptions import ReportGenerationError, ReportTemplateError | ||
|
|
||
|
|
||
| def generate_transactions_pdf(event: dict, logger: Logger) -> bytes: | ||
| current_dir = os.path.dirname(os.path.abspath(__file__)) | ||
| env = Environment( | ||
| loader=FileSystemLoader(current_dir), | ||
| autoescape=select_autoescape(["html", "xml"]), | ||
| ) | ||
|
|
||
| try: | ||
| template = env.get_template("template.html") | ||
| except TemplateNotFound as e: | ||
| logger.error("Template 'template.html' not found") | ||
| raise ReportTemplateError("Missing template: template.html") from e | ||
|
|
||
| html_out = template.render( | ||
| accountId=event["accountId"], | ||
| statementPeriod=event["statementPeriod"], | ||
| transactions=event["transactions"], | ||
| accountBalance=event["accountBalance"], | ||
| generationDate=datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC"), | ||
| ) | ||
|
|
||
| pdf_buffer = io.BytesIO() | ||
| pisa_status = pisa.CreatePDF(io.StringIO(html_out), dest=pdf_buffer) | ||
|
|
||
| if pisa_status.err: | ||
| logger.error("xhtml2pdf failed to generate PDF") | ||
| raise ReportGenerationError("Error generating PDF") | ||
|
|
||
| pdf_buffer.seek(0) | ||
| pdf_bytes = pdf_buffer.getvalue() | ||
|
|
||
| logger.debug("PDF generated (%d bytes).", len(pdf_bytes)) | ||
| return pdf_bytes |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.