Download script is changed to add all quarters of data

[Krishnam24maheshwari]

No description provided.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[gemini-code-assist]

Code Review

This pull request refactors the Brazil SIDRA IBGE download script, replacing the Selenium-based browser automation with direct API calls using the requests library and reshaping the data into Excel files using pandas. The review feedback recommends improving performance, security, and robustness by using a global requests.Session with connection pooling and retries, re-enabling SSL verification, and adding explicit guard checks to safely parse the API responses.

[gemini-code-assist]

Using requests.get repeatedly in a loop (via fetch_aggregate_series) is inefficient because it opens and closes a new connection for every request, performing a full TCP and SSL handshake each time. Since this script makes hundreds of API calls, using a global requests.Session will enable connection pooling and keep-alive, significantly improving performance.

Additionally, configuring retries on this session will satisfy the requirement for robust downloading from unstable external sources, as specified in the general rules.

Finally, disabling SSL verification (verify=False) and suppressing warnings is a security risk as it exposes the script to Man-in-the-Middle (MitM) attacks. We have enabled SSL verification in the suggested session configuration.

import os
import time
import requests
import urllib3
import pandas as pd
from absl import app, logging, flags
from pathlib import Path
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

# Configure a global session with retries and connection pooling
def get_robust_session():
    session = requests.Session()
    retries = Retry(
        total=10,
        backoff_factor=1,
        status_forcelist=[500, 502, 503, 504],
        raise_on_status=False
    )
    session.mount("https://", HTTPAdapter(max_retries=retries))
    return session

SESSION = get_robust_session()

References

1. When downloading data from unstable or unreliable external sources, configure a higher number of retries (e.g., tries=10) to ensure download success, even if it results in a long cumulative wait time.

[gemini-code-assist]

Use the global SESSION object to perform the HTTP request. This enables connection pooling, automatic retries, and keeps SSL verification enabled.

Suggested change

	res = requests.get(url, headers=headers, verify=False, timeout=30)
	res = SESSION.get(url, headers=headers, timeout=30)

[gemini-code-assist]

Use the global SESSION object to perform the HTTP request. This enables connection pooling, automatic retries, and keeps SSL verification enabled.

Suggested change

	r = requests.get(url, headers=headers, verify=False, timeout=30)
	r = SESSION.get(url, headers=headers, timeout=30)

[gemini-code-assist]

To prevent potential KeyError or TypeError when the API returns an unexpected dictionary or non-list response, explicitly verify that data is a non-empty list before accessing data[0]. This aligns with the preference for explicit guard checks over broad exception handling for nested data structures.

Suggested change

	data = r.json()
	if not data or "resultados" not in data[0]:
	return {}
	data = r.json()
	if not isinstance(data, list) or not data or "resultados" not in data[0]:
	return {}

References

1. Prefer explicit guard checks over broad exception control flow (try...except) for handling nested data structures, as they are easier to reason about.