Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add simple RMGIpc for remage-cpp -> python intercom #246

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

ManuelHu
Copy link
Collaborator

@ManuelHu ManuelHu commented Jan 28, 2025

this is very much WIP and needs good testing before releasing

this is a simple IPC mechanism. Its main features:

  • one-way communication of unicode strings from the C++ side to python
  • asynchronous messages do not get an acknowledgement back
    (this is meant for informational messages that the python code needs after running remage-cpp, e.g. output file post-processing).
  • synchronous messages require an acknowledgement signal from python. for this, the POSIX signal SIGUSR2 is used
    (this is meant for actions that python code should perform synchronously, e.g. working on input files)

by using this mechanism we can avoid re-implementing ugly logic in the python wrapper (e.g. guessing output file names, etc.) and get that data just form the C++ application.

this PR does not implement any actions on the python side yet. But with minor additions, this could be used for #190

  • the biggest changes will be on the pydataobj side, as there is no way to call the functionality of lh5concat from python code...
  • also, as @tdixon97 noted: lh5concat does currently load all files into memory, and does not use an iterator with smaller buffer size. This might not be what we want

@ManuelHu ManuelHu force-pushed the ipc branch 3 times, most recently from 782ad45 to 617e884 Compare January 28, 2025 10:22
@tdixon97
Copy link
Collaborator

@ManuelHu I guess it wouldnt be a big modification to make lh5concat iterate? Seems worthwhile also for other applications

@gipert
Copy link
Member

gipert commented Jan 28, 2025

I think lh5concat changes shouldn't be hard to implement

@ManuelHu ManuelHu force-pushed the ipc branch 6 times, most recently from 4bbbca2 to 65f2c31 Compare January 30, 2025 07:38
@ManuelHu ManuelHu marked this pull request as ready for review January 30, 2025 10:42
@ManuelHu
Copy link
Collaborator Author

this should now be mostly ready. I would like to implement any actual merging in a follow-up PR to keep apart the infrastructure work and the actual features.

@tdixon97
Copy link
Collaborator

tdixon97 commented Feb 2, 2025

I don't understand what it is, but looks good

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants