Exception in asyncio and memory leak

[hpatriarche]

Using yocto sumo version with not stable Hawkbit server we got this following issue:

Nov 26 15:59:36  rauc-hawkbit-client[490]: 2018-11-26 15:59:36 INFO     Starting bundle download
Nov 26 16:01:17  rauc-hawkbit-client[490]: 2018-11-26 16:01:17 ERROR    Polling failed with an unexpected exception:
Nov 26 16:01:17  rauc-hawkbit-client[490]: Traceback (most recent call last):
Nov 26 16:01:17  rauc-hawkbit-client[490]:   File "/usr/lib/python3.5/site-packages/rauc_hawkbit/rauc_dbus_ddi_client.py", line 151, in start_polling
Nov 26 16:01:17  rauc-hawkbit-client[490]:     await self.poll_base_resource()
Nov 26 16:01:17  rauc-hawkbit-client[490]:   File "/usr/lib/python3.5/site-packages/rauc_hawkbit/rauc_dbus_ddi_client.py", line 310, in poll_base_resource
Nov 26 16:01:17  rauc-hawkbit-client[490]:     await self.process_deployment(base)
Nov 26 16:01:17  rauc-hawkbit-client[490]:   File "/usr/lib/python3.5/site-packages/rauc_hawkbit/rauc_dbus_ddi_client.py", line 238, in process_deployment
Nov 26 16:01:17  rauc-hawkbit-client[490]:     await self.download_artifact(action_id, download_url, md5_hash)
Nov 26 16:01:17  rauc-hawkbit-client[490]:   File "/usr/lib/python3.5/site-packages/rauc_hawkbit/rauc_dbus_ddi_client.py", line 271, in download_artifact
Nov 26 16:01:17  rauc-hawkbit-client[490]:     .artifacts[filename](self.bundle_dl_location)
Nov 26 16:01:17  rauc-hawkbit-client[490]:   File "/usr/lib/python3.5/site-packages/rauc_hawkbit/ddi/softwaremodules.py", line 19, in __call__
Nov 26 16:01:17  rauc-hawkbit-client[490]:     filename=self.file_name)
Nov 26 16:01:17  rauc-hawkbit-client[490]:   File "/usr/lib/python3.5/site-packages/rauc_hawkbit/ddi/client.py", line 200, in get_binary_resource
Nov 26 16:01:17  rauc-hawkbit-client[490]:     timeout=timeout)
Nov 26 16:01:17  rauc-hawkbit-client[490]:   File "/usr/lib/python3.5/site-packages/rauc_hawkbit/ddi/client.py", line 234, in get_binary
Nov 26 16:01:17  rauc-hawkbit-client[490]:     chunk = await resp.content.read(chunk_size)
Nov 26 16:01:17  rauc-hawkbit-client[490]:   File "/usr/lib/python3.5/site-packages/aiohttp/streams.py", line 542, in read
Nov 26 16:01:17  rauc-hawkbit-client[490]:     return (yield from super().read(n))
Nov 26 16:01:17  rauc-hawkbit-client[490]:   File "/usr/lib/python3.5/site-packages/aiohttp/streams.py", line 298, in read
Nov 26 16:01:17  rauc-hawkbit-client[490]:     yield from self._wait('read')
Nov 26 16:01:17  rauc-hawkbit-client[490]:   File "/usr/lib/python3.5/site-packages/aiohttp/streams.py", line 226, in _wait
Nov 26 16:01:17  rauc-hawkbit-client[490]:     yield from waiter
Nov 26 16:01:17  rauc-hawkbit-client[490]:   File "/usr/lib/python3.5/asyncio/futures.py", line 381, in __iter__
Nov 26 16:01:17  rauc-hawkbit-client[490]:     yield self  # This tells Task to wait for completion.
Nov 26 16:01:17  rauc-hawkbit-client[490]:   File "/usr/lib/python3.5/asyncio/tasks.py", line 310, in _wakeup
Nov 26 16:01:17  rauc-hawkbit-client[490]:     future.result()
Nov 26 16:01:17  rauc-hawkbit-client[490]:   File "/usr/lib/python3.5/asyncio/futures.py", line 294, in result
Nov 26 16:01:17  rauc-hawkbit-client[490]:     raise self._exception
Nov 26 16:01:17  rauc-hawkbit-client[490]: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Nov 26 16:01:17  rauc-hawkbit-client[490]: 2018-11-26 16:01:17 INFO     Retry will happen in 60 seconds

And moreover memory use by python3 process increase at each try ...

[hpatriarche]

It looks similar to what Livio described in #8
but even with using aiohttp 3.3.2 + his patch. It is still failing:

2018-11-27 19:45:20 ERROR    Polling failed with an unexpected exception:
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/rauc_hawkbit/rauc_dbus_ddi_client.py", line 151, in start_polling
    await self.poll_base_resource()
  File "/usr/lib/python3.5/site-packages/rauc_hawkbit/rauc_dbus_ddi_client.py", line 317, in poll_base_resource
    await self.process_deployment(base)
  File "/usr/lib/python3.5/site-packages/rauc_hawkbit/rauc_dbus_ddi_client.py", line 245, in process_deployment
    await self.download_artifact(action_id, download_url, md5_hash)
  File "/usr/lib/python3.5/site-packages/rauc_hawkbit/rauc_dbus_ddi_client.py", line 278, in download_artifact
    .artifacts[filename](self.bundle_dl_location)
  File "/usr/lib/python3.5/site-packages/rauc_hawkbit/ddi/softwaremodules.py", line 19, in __call__
    filename=self.file_name)
  File "/usr/lib/python3.5/site-packages/rauc_hawkbit/ddi/client.py", line 198, in get_binary_resource
    return await self.get_binary(url, dl_location, mime, 3600)
  File "/usr/lib/python3.5/site-packages/rauc_hawkbit/ddi/client.py", line 232, in get_binary
    chunk, _ = await resp.content.readchunk()
  File "/usr/lib/python3.5/site-packages/aiohttp/streams.py", line 347, in readchunk
    await self._wait('readchunk')
  File "/usr/lib/python3.5/site-packages/aiohttp/streams.py", line 250, in _wait
    await waiter
  File "/usr/lib/python3.5/asyncio/futures.py", line 381, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 310, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 294, in result
    raise self._exception
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

[Bastian-Krause]

There are quite some issues open for ClientPayloadErrors on aiohttp. I'd guess either something changed in hawkBit's HTTP server or you are using some kind of proxy. Please verify that downloading the artifact works with a minimalistic aiohttp client example.

[hpatriarche]

Well after some investigation and testing the issue is related to file transfer which is not correctly resume when server decide to stop the transfer.
We need to handle resume like wget command do.

Attach a patch proposal to handle that.
0003-Manage-connection-resume.txt

[Bastian-Krause]

Could you provide this as a PR, please (based on #8 if necessary)?