fix: retry result uploads instead of silently losing them (#504)

qase-python-commons/README.md

@@ -30,6 +30,7 @@ Core library for all Qase Python reporters. Contains the complete configuration
   - [Status Filtering](#status-filtering)
   - [External Links](#external-links)
   - [Test Run Configurations](#test-run-configurations)
+  - [Upload Reliability](#upload-reliability)
 
 ---
 
@@ -102,6 +103,9 @@ The reporter mode is set via the `mode` option:
 |-------------|-------------|---------------------|---------|----------|
 | API token | `testops.api.token` | `QASE_TESTOPS_API_TOKEN` | — | Yes* |
 | API host | `testops.api.host` | `QASE_TESTOPS_API_HOST` | `qase.io` | No |
+| Request timeout, seconds | `testops.api.timeout` | `QASE_TESTOPS_API_TIMEOUT` | `30` | No |
+| Upload attempts | `testops.api.retries` | `QASE_TESTOPS_API_RETRIES` | `3` | No |
+| Retry backoff base, seconds | `testops.api.retryBackoff` | `QASE_TESTOPS_API_RETRY_BACKOFF` | `2` | No |
 | Project code | `testops.project` | `QASE_TESTOPS_PROJECT` | — | Yes* |
 | Test run ID | `testops.run.id` | `QASE_TESTOPS_RUN_ID` | — | No |
 | Test run title | `testops.run.title` | `QASE_TESTOPS_RUN_TITLE` | `Automated run <date>` | No |
@@ -261,6 +265,9 @@ export QASE_TESTOPS_API_TOKEN="<token>"
 export QASE_TESTOPS_PROJECT="DEMO"
 export QASE_TESTOPS_RUN_TITLE="Automated Run"
 export QASE_TESTOPS_RUN_COMPLETE="true"
+export QASE_TESTOPS_API_TIMEOUT="30"
+export QASE_TESTOPS_API_RETRIES="3"
+export QASE_TESTOPS_API_RETRY_BACKOFF="2"
 
 # Pytest
 export QASE_PYTEST_CAPTURE_LOGS="true"
@@ -374,6 +381,44 @@ Creates or finds configurations in Qase TestOps:
 }
 ```
 
+### Upload Reliability
+
+Results are uploaded in batches from background threads. A batch that fails on
+a transient error is retried rather than dropped.
+
+| Setting | Meaning |
+|---------|---------|
+| `testops.api.timeout` | Per-request timeout in seconds. Without it a stalled connection blocks the session at teardown. |
+| `testops.api.retries` | Total attempts per batch, not retries on top of the first. `3` means three tries; `0` sends once and never retries. |
+| `testops.api.retryBackoff` | Base of the exponential delay: attempt *n* waits `retryBackoff ** n` seconds. |
+
+Retried: connection resets, timeouts and other transport failures, plus HTTP
+408, 429 and 5xx. Not retried: 400, 401, 403, 404, 413, 422 and 507 — a second
+attempt fails identically and only adds load.
+
+When a 429 carries a `Retry-After` header, that value replaces the computed
+backoff. Qase sends roughly 60 seconds, so a run that hits the rate limit takes
+longer to finish rather than losing the batch.
+
+**If a batch cannot be delivered after all attempts**, the reporter logs an
+error naming how many results were lost and **does not mark the run complete**.
+An open run is the signal that its data is incomplete; a completed run over
+partial results would look trustworthy and not be. In `testops_multi` mode this
+is per project — one project's failure does not stop the others completing.
+
+```json
+{
+  "testops": {
+    "api": {
+      "token": "<token>",
+      "timeout": 30,
+      "retries": 3,
+      "retryBackoff": 2
+    }
+  }
+}
+```
+
 ---
 
 ## Requirements

qase-python-commons/changelog.md

@@ -1,3 +1,13 @@
+# qase-python-commons@5.1.4
+
+## What's new
+
+- Fixed silent loss of test results when a batch upload fails ([#504](https://github.com/qase-tms/qase-python/issues/504)). The reporter discarded its only copy of a batch when it started the upload thread rather than when the server confirmed it, so a connection reset made those results cease to exist while the run was still marked complete and the process exited 0. Failed batches are now retried; if a batch still cannot be delivered, the reporter logs an error naming how many results were lost and leaves the run open instead of completing it. In `testops_multi` mode this is tracked per project, so one project's failure does not stop the others completing.
+- `ResultCreate.id` is now sent on bulk upload. The v2 API uses it as an idempotency key, and the value was already generated on every result but dropped when building the payload. Without it a retried batch would create duplicate results.
+- Added `testops.api.timeout` (default `30`), `testops.api.retries` (default `3`) and `testops.api.retryBackoff` (default `2`), with `QASE_TESTOPS_API_TIMEOUT`, `QASE_TESTOPS_API_RETRIES` and `QASE_TESTOPS_API_RETRY_BACKOFF` overrides. Retries cover transport failures and HTTP 408/429/5xx, honour `Retry-After`, and never fire on 400/401/403/404/413/422/507. `retries` counts total attempts, so `0` sends once without retrying.
+- Set an explicit request timeout on result uploads. There was none, so a connection that stalled rather than failed hung the session at teardown until CI killed the job.
+- Pinned urllib3's own retry policy off (`Configuration.retries = 0`). It was left unset, so urllib3's defaults applied and would have multiplied with the new application-level retry.
+
 # qase-python-commons@5.1.3
 
 ## What's new

qase-python-commons/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "qase-python-commons"
-version = "5.1.3"
+version = "5.1.4"
 description = "A library for Qase TestOps and Qase Report"
 readme = "README.md"
 authors = [{name = "Qase Team", email = "support@qase.io"}]

qase-python-commons/src/qase/commons/client/api_v1_client.py

@@ -26,6 +26,9 @@ def __init__(self, config: QaseConfig, logger: Logger):
             configuration = Configuration()
             configuration.api_key['TokenAuth'] = self.config.testops.api.token
             configuration.ssl_ca_cert = certifi.where()
+            # See the note in ApiV2Client: urllib3's default retry policy is
+            # pinned off so it cannot multiply with the application-level one.
+            configuration.retries = 0
             host = self.config.testops.api.host
             if host == 'qase.io':
                 configuration.host = f'https://api.{host}/v1'

qase-python-commons/src/qase/commons/client/api_v2_client.py

@@ -36,6 +36,11 @@ def __init__(self, config: QaseConfig, logger: Logger, host_data: Optional[HostD
             configuration = Configuration()
             configuration.api_key['TokenAuth'] = self.config.testops.api.token
             configuration.ssl_ca_cert = certifi.where()
+            # Pin urllib3's own retry policy off: the reporter retries at the
+            # application level, and leaving this at None lets urllib3's
+            # defaults multiply with ours -- a 3-attempt policy would issue
+            # 9 requests, the opposite of what is wanted on a 429.
+            configuration.retries = 0
             host = self.config.testops.api.host
             if host == 'qase.io':
                 configuration.host = f'https://api.{host}/v2'
@@ -131,7 +136,8 @@ def send_results(self, project_code: str, run_id: str, results: []) -> None:
         run_id_int = int(run_id) if isinstance(run_id, str) else run_id
         self.logger.log_debug(f"Sending results for run {run_id_int}: {results_to_send}")
         api_results.create_results_v2(project_code, run_id_int,
-                                      create_results_request_v2=CreateResultsRequestV2(results=results_to_send))
+                                      create_results_request_v2=CreateResultsRequestV2(results=results_to_send),
+                                      _request_timeout=self.config.testops.api.timeout)
         self.logger.log_debug(f"Results for run {run_id_int} sent successfully")
 
     def _prepare_result(self, project_code: str, result: Result) -> ResultCreate:
@@ -158,6 +164,7 @@ def _prepare_result(self, project_code: str, result: Result) -> ResultCreate:
                 result.params[key] = "empty"
 
         result_model_v2 = ResultCreate(
+            id=result.id,
             title=result.get_title(),
             signature=result.signature,
             testops_ids=result.get_testops_ids(),

qase-python-commons/src/qase/commons/config.py

@@ -109,6 +109,20 @@ def __load_file_config(self):
                                 self.config.testops.api.set_token(
                                     api.get("token"))
 
+                            if api.get("timeout"):
+                                self.config.testops.api.set_timeout(
+                                    api.get("timeout"))
+
+                            # `is not None`, not truthiness: 0 is a valid value
+                            # for both, meaning "no retries" / "no backoff".
+                            if api.get("retries") is not None:
+                                self.config.testops.api.set_retries(
+                                    api.get("retries"))
+
+                            if api.get("retryBackoff") is not None:
+                                self.config.testops.api.set_retry_backoff(
+                                    api.get("retryBackoff"))
+
                         if testops.get("project"):
                             self.config.testops.set_project(
                                 testops.get("project"))
@@ -288,6 +302,15 @@ def __load_env_config(self):
                 if key == 'QASE_TESTOPS_API_TOKEN':
                     self.config.testops.api.set_token(value)
 
+                if key == 'QASE_TESTOPS_API_TIMEOUT':
+                    self.config.testops.api.set_timeout(value)
+
+                if key == 'QASE_TESTOPS_API_RETRIES':
+                    self.config.testops.api.set_retries(value)
+
+                if key == 'QASE_TESTOPS_API_RETRY_BACKOFF':
+                    self.config.testops.api.set_retry_backoff(value)
+
                 if key == 'QASE_TESTOPS_PROJECT':
                     self.config.testops.set_project(value)
 

qase-python-commons/src/qase/commons/models/config/api.py

@@ -4,12 +4,36 @@
 class ApiConfig(BaseModel):
     token: str = None
     host: str = None
+    timeout: int = None
+    retries: int = None
+    retry_backoff: int = None
 
     def __init__(self):
         self.host = "qase.io"
+        self.timeout = 30
+        self.retries = 3
+        self.retry_backoff = 2
 
     def set_token(self, token: str):
         self.token = token
 
     def set_host(self, host: str):
         self.host = host
+
+    def set_timeout(self, timeout: int):
+        timeout = int(timeout)
+        if timeout <= 0:
+            raise ValueError("API timeout should be greater than 0")
+        self.timeout = timeout
+
+    def set_retries(self, retries: int):
+        retries = int(retries)
+        if retries < 0:
+            raise ValueError("API retries should be 0 or greater")
+        self.retries = retries
+
+    def set_retry_backoff(self, retry_backoff: int):
+        retry_backoff = int(retry_backoff)
+        if retry_backoff < 0:
+            raise ValueError("API retry backoff should be 0 or greater")
+        self.retry_backoff = retry_backoff

qase-python-commons/src/qase/commons/reporters/testops.py

@@ -8,6 +8,7 @@
 from ..client.base_api_client import BaseApiClient
 from ..models import Result
 from ..models.config.qaseconfig import QaseConfig
+from ..retry import send_with_retry
 
 DEFAULT_BATCH_SIZE = 200
 DEFAULT_THREAD_COUNT = 4
@@ -36,6 +37,8 @@ def __init__(self, config: QaseConfig, logger: Logger, client: BaseApiClient) ->
         self.send_semaphore = threading.Semaphore(DEFAULT_THREAD_COUNT)  # Semaphore to limit concurrent sends
         self.lock = threading.Lock()
         self.count_running_threads = 0
+        # Results in batches that could not be delivered after all retries.
+        self.lost_results = 0
 
         environment = self.config.environment
         if environment:
@@ -70,15 +73,28 @@ def __init__(self, config: QaseConfig, logger: Logger, client: BaseApiClient) ->
 
     def _send_results_threaded(self, results):
         try:
-            self.client.send_results(self.project_code, self.run_id, results)
+            send_with_retry(
+                lambda: self.client.send_results(self.project_code, self.run_id, results),
+                attempts=self.config.testops.api.retries,
+                backoff=self.config.testops.api.retry_backoff,
+                logger=self.logger,
+            )
             with self.lock:
                 self.processed.extend(results)
         except Exception as e:
+            # Account for the batch here rather than re-raising. The raise only
+            # ever surfaced as a PytestUnhandledThreadExceptionWarning that
+            # nothing acted on, which is how the loss stayed invisible.
             with self.lock:
-                self.logger.log(f"Error at sending results for run {self.run_id}: {e}", "error")
-            raise  # Re-raise the exception to be caught by the thread handler
+                self.lost_results += len(results)
+                self.logger.log(
+                    f"Failed to send {len(results)} results for run {self.run_id} "
+                    f"after {self.config.testops.api.retries} attempt(s): {e}",
+                    "error",
+                )
         finally:
-            self.count_running_threads -= 1
+            with self.lock:
+                self.count_running_threads -= 1
             self.send_semaphore.release()  # Release semaphore whether success or exception
 
     def _send_results(self) -> None:
@@ -101,7 +117,8 @@ def _send_results(self) -> None:
             if results_to_send:
                 # Acquire semaphore before starting the send operation
                 self.send_semaphore.acquire()
-                self.count_running_threads += 1
+                with self.lock:
+                    self.count_running_threads += 1
 
                 # Start a new thread for sending results
                 send_thread = threading.Thread(target=self._send_results_threaded, args=(results_to_send,))
@@ -136,7 +153,17 @@ def complete_run(self) -> None:
 
         while self.count_running_threads > 0:
             time.sleep(DEFAULT_THREAD_POLL_INTERVAL)
-
+
+        if self.lost_results > 0:
+            # Leaving the run open is the signal. A run marked complete over
+            # partial data looks trustworthy and is not.
+            self.logger.log(
+                f"{self.lost_results} result(s) were not delivered to Qase. "
+                f"The run will not be marked complete so the gap stays visible.",
+                "error",
+            )
+            return
+
         if self.complete_after_run:
             self.logger.log_debug("Completing run")
             self.client.complete_run(self.project_code, self.run_id)
@@ -158,7 +185,7 @@ def complete_worker(self) -> None:
         if len(self.results) > 0:
             self._send_results()
         while self.count_running_threads > 0:
-            pass
+            time.sleep(DEFAULT_THREAD_POLL_INTERVAL)
         self.logger.log_debug("Worker completed")
 
     def add_result(self, result: Result) -> None: