You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Update : Async does not really help as the task is not I/O bound but CPU bound after timing the program run time. Will have to use multiprocessing for multiple files redaction.
python3 pyredactkit.py ip_test.txt 39.31s user 0.16s system 100% cpu 39.412 total
More update after running benchmarks. Reading and Writing files to disk is already non blocking. The "concurrency" only comes in handy when you're doing requests over the network.
For running 5 text files
With threading
poetry run prk logtest 39.58s user 1.15s system 97% cpu 41.678 total
Without threading
poetry run prk logtest 20.05s user 0.43s system 93% cpu 21.897 total
Checklist
master
branch of the latest code.Is your feature request related to a problem? Please describe
For large multiple files, the process time takes (n number of files * time).
Describe the solution you'd like
Explore into either async or multi processing for handling multiple files in parallel.
Describe alternatives you've considered
Threading is one other option
Additional context
The text was updated successfully, but these errors were encountered: