Commit Graph

2 Commits

Author SHA1 Message Date
grekiki 398796a010 FileReader: cache files locally (#2170)
* framework for updates

* File caching

* update .gitignore

* update .gitignore

* Store in home tmp directory

* fix bugs in old code, add tests

* add test

* revert bug

* Fixed some bugs, more testing

* doing some style fixes

* Update caching with lenghts

* doing some style fixes

* doing some style fixes

* doing some style fixes

* subrepositories

* try to fix the tests

* change submodules back

* doing some style fixes

* if we don't cache, we don't cache length either

* fix curls

* make sure tests run in ci

* Cleanup tests

* Use int for cache flag

* fix linter

Co-authored-by: Willem Melching <willem.melching@gmail.com>
old-commit-hash: 05b64257dfac3e240135e9f40eb5ebaa8f97fdcb
2020-09-17 13:03:34 +02:00
Willem Melching 9c291968f6 Speedup URLFile (#1888)
* add parallel download support to URLFile

* make python 3.8 happy

* Fix chunk size

* Automatic number of threads

* No daemon threads in unlogger

* Cache length

* dont touch old filereader

* Remove debug info

* remove debug script

* Ignore type
old-commit-hash: c70700758d9e3227a157d2e9fdcb9d3204b433c2
2020-07-20 17:10:08 +02:00