Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Promote Supermask out of prototype #1729

Merged
merged 11 commits into from
Feb 18, 2025
Merged

Promote Supermask out of prototype #1729

merged 11 commits into from
Feb 18, 2025

Conversation

jcaip
Copy link
Contributor

@jcaip jcaip commented Feb 18, 2025

This PR promotes Supermask and block sparsity from prototype -> torchao.sparsity, instead of the apply_supermask function which was previously closely coupled with SAM.

It adds a new public API for SupermaskLinear, which users can use to add Supermask to their models for training with

sparsify_(model, lambda x: SupermaskLinear.from_linear(x, block_size=64, sparsity_level=0.9)

To accelerate for inference, we convert the SupermaskLinear model back into a nn.Linear, which simplifies the Supermask logic:

sparsify_(model, lambda x: SupermaskLinear.to_linear(x, sparsity_level=0.9)

bc-breaking

The previous prototype APIs, torchao.sparsity.prototype.superblock.supermask and torchao.prototype.sparsity.superblock.supermask have been deprecated.

You can use torchao.sparsity.supermask instead.

Copy link

pytorch-bot bot commented Feb 18, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1729

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 1 Pending

As of commit 343f3fe with merge base 7b37eb0 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@jcaip jcaip marked this pull request as ready for review February 18, 2025 21:21
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 18, 2025
@jcaip jcaip added sparsity topic: bc-breaking Use this tag if this PR breaks backward compatibility labels Feb 18, 2025
@jcaip jcaip merged commit 79ac44e into main Feb 18, 2025
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. sparsity topic: bc-breaking Use this tag if this PR breaks backward compatibility
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants