Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Avoid tracking small allocations by setting
memThreshold
In order to avoid running out of memory and crashing the server, Arkouda uses `memTrack` to track memory allocations and raise a client error when a large allocation would likely exceed the available memory. Recently, we discovered some performance overheads from this tracking while doing highly concurrent short-lived allocations. The `memTrack` flag is implemented with a global lock to protect access to a hash table, which is serializing and really hurting the performance of concurrent allocs. To limit the impact of this, use `memThreshold=1M` to only track larger allocations and skip skip tracking smaller ones. Using `memThreshold` did not have quite the performance boost I was expecting at first with Chapel 1.25 because we didn't have a way to do threshold checks on free. That was added in chapel-lang/chapel 18465. Here's an allocation micro-benchmark that shows the performance on a single node with 128-Rome CPUs: ```chpl use Time; config const trials = 1_000_000; var t: Timer; t.start(); coforall 1..here.maxTaskPar do for i in 1..trials do var s = i:string; writeln(t.elapsed()); ``` | config | Time | | ------------------- | ------: | | w/o memTrack | 0.19s | | w/ memTrack | 144.50s | | w/ threshold 1.25.0 | 33.06s | | w/ threshold main | 0.22s | So this should benefit code that has highly concurrent allocations like casting to strings and currently regex operations (though we're working on optimizing the regex allocations out.) Part of 675 Part of 929
- Loading branch information