Push after running neuron_parallel_compile
#354
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
dacorvo
reviewed
Nov 24, 2023
optimum/neuron/utils/misc.py
Outdated
| from .require_utils import requires_safetensors | ||
|
|
||
|
|
||
| logger = logging.get_logger() | ||
|
|
||
|
|
||
| def should_log() -> bool: |
There was a problem hiding this comment.
This is very specific to neuronx_distributed, and not really related to logging. Can you move it under the neuronx_distributed directory, and choose a name that clearly states that this verifies we are in the process of the first worker (that's what I understood).
There was a problem hiding this comment.
No it's linked to the overall torch.distributed, it used everywhere, like in the Trainer and everything. Will renaming it.
dacorvo
approved these changes
Nov 24, 2023
optimum/neuron/utils/misc.py
Outdated
| @@ -45,7 +45,7 @@ | |||
| logger = logging.get_logger() | |||
|
|
|||
|
|
|||
| def should_log() -> bool: | |||
| def should_current_worker_log() -> bool: | |||
There was a problem hiding this comment.
It still has nothing to do with log. I would call it is_main_worker, wdyt ?
There was a problem hiding this comment.
Alright, doing it.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for pushing to the cache repository on the Hugging Face Hub after using
neuron_paralell_compile.Basically, when running a training after doing
neuron_parallel_compile, all the compilation files will be pushed to the Hub before starting actual training.cc @5cp