You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
I was going to spend a couple of days adding support for loading different file types using rosetasciio directly into py4DSTEM. Is there is anything that I might need to consider before doing this.
Describe the solution you'd like
Most of the complications are going to be limiting what filetypes should be allowed to be loaded and which filetypes are not supported. Currently Rosettasciio requires dask as a dependancy, is this an issue? I could potentially remove it as a requirement and only allow for non-lazy loading if this is something that you would prefer not to add. Do you have a current process for handling memory mapped arrays, or is treating them like numpy arrays and then having numpy automatically convert to in memory arrays fine.
Additional context
This should be a good start to understanding how to generalize the process of loading/ transferring data. I think that ultimately we need to have a .yaml file for each file format which details if it outputs 4D Data, and what metadata it has.
Edit:
It seems like there is already a quite nice version here for loading using memmap vs RAM that we could extend.
Thanks @CSSFrancis , this sounds great! Dask dependency is not a problem. And yup, you found our memory mapping code. Eventually having a .yaml per file format is a nice idea too. Please let us know if you have other questions :)
Is your feature request related to a problem? Please describe.
I was going to spend a couple of days adding support for loading different file types using rosetasciio directly into py4DSTEM. Is there is anything that I might need to consider before doing this.
Describe the solution you'd like
Most of the complications are going to be limiting what filetypes should be allowed to be loaded and which filetypes are not supported. Currently Rosettasciio requires dask as a dependancy, is this an issue? I could potentially remove it as a requirement and only allow for non-lazy loading if this is something that you would prefer not to add. Do you have a current process for handling memory mapped arrays, or is treating them like numpy arrays and then having numpy automatically convert to in memory arrays fine.
Additional context
This should be a good start to understanding how to generalize the process of loading/ transferring data. I think that ultimately we need to have a .yaml file for each file format which details if it outputs 4D Data, and what metadata it has.
Edit:
It seems like there is already a quite nice version here for loading using memmap vs RAM that we could extend.
https://github.com/py4dstem/py4DSTEM/blob/966a41c7611161d4b3d8d082af5a145be0653b76/py4DSTEM/io/importfile.py#L17C5-L23
The text was updated successfully, but these errors were encountered: