Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for T5-11b T5-XXL + TP #317

Closed
samir-souza opened this issue Nov 10, 2023 · 11 comments · Fixed by #697
Closed

Support for T5-11b T5-XXL + TP #317

samir-souza opened this issue Nov 10, 2023 · 11 comments · Fixed by #697
Assignees
@JingyaHuang

Comments

@samir-souza
Copy link

The XXL version of the model for Inference is being asked by many customers. Given the size of the model, only through TP it would be possible to load the model.
@JingyaHuang JingyaHuang self-assigned this Nov 10, 2023
@michaelbenayoun
Copy link
Member

It's in the roadmap: #267 will add general support for export and inference using neuronx_distributed and we will add support for TP in next PRs.

@HuggingFaceDocBuilderDev
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

1 similar comment
@HuggingFaceDocBuilderDev
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

@JingyaHuang JingyaHuang reopened this Apr 23, 2024
@JingyaHuang
Copy link
Collaborator

T5 with TP support is not supported yet, I'm still waiting for the help from the Annapurna team on the issue aws-neuron/aws-neuron-sdk#851

@HuggingFaceDocBuilderDev
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

5 similar comments
@HuggingFaceDocBuilderDev
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

@HuggingFaceDocBuilderDev
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

@HuggingFaceDocBuilderDev
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

@HuggingFaceDocBuilderDev
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

@HuggingFaceDocBuilderDev
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

@JingyaHuang JingyaHuang mentioned this issue Sep 19, 2024
Merged
8 tasks
@HuggingFaceDocBuilderDev
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JingyaHuang JingyaHuang

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add tensor parallel support to T5 via NxD huggingface/optimum-neuron
4 participants
@michaelbenayoun @samir-souza @JingyaHuang @HuggingFaceDocBuilderDev