Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rethink about stream decoding #18

Open
dogtopus opened this issue Mar 6, 2023 · 2 comments
Open

Rethink about stream decoding #18

dogtopus opened this issue Mar 6, 2023 · 2 comments
Labels
enhancement New feature or request
Milestone

Comments

@dogtopus
Copy link
Owner

dogtopus commented Mar 6, 2023

Some old version of MiniPB uses stream decoding exclusively. However due to how Protobuf handles repeated non-repeated fields (i.e. multiple fields with the same ID while the field is non-repeated) this was changed to pre-indexing the whole message and decode in one go.

We might still be able to bring stream decoding back by making the decode states read-write and return a snapshot of it when the decoding is done (by reaching an end-of-message marker either set by the user or naturally occurring).

This could also make implementing #11 easier and in a much cleaner way (instead of committing the states to a list/object, we just yield the state changes the first time we see them. It would be user's responsibility to only use the last one that pops up).

@dogtopus dogtopus added the enhancement New feature or request label Mar 6, 2023
@dogtopus dogtopus added this to the v2.1.0 milestone Mar 6, 2023
@ghost
Copy link

ghost commented Jul 5, 2023

I am reversing into a very sloppy work around for accomplishing this on a streaming file for GTFS-RT (transit trip updates). Not sure if I have technical skill to help develop this but wanted to say it would open a lot of doors for me and perhaps others on their projects. This library is so extremely useful and light weight already, but being able to use it with 192kb of RAM is still a challenge.

@ghost
Copy link

ghost commented Jan 2, 2024

I got a device that has enough RAM to decode in one go. Adafruit Matrix Portal S3 which as 2MB vs 192KB. Unfortunately, the memory on device gets too fragmented to store results so back to square one. (3.5 kb allocations fail while 300+ kb memory is free).

Do you have any tips on best places place garbage collection calls in minipb.py to keep the memory?
Or any decent ways in theory I can evalate and throw out messages that aren't of interest? I know this is likely the same as a streaming feature you mentioned in the issue body and likely not so simple.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant