-
Notifications
You must be signed in to change notification settings - Fork 113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bump protobuf to 3.10.0 and apply fixes for --incompatible_load_{java|proto|python}_rules_from_bzl #413
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for working on this!
Why did you add java/io/bazel/rules/closure/Webpath.class
, closure/v3.8.0.tar.gz
and closure/v3.8.0.tar.gz.1
? Was it an accident, or am I missing something?
@@ -1,11 +1,12 @@ | |||
# DO NOT EDIT -- bazel run //closure/library:regenerate -- "$PWD" | |||
|
|||
load("@rules_python//python:defs.bzl", "py_binary") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add this load here [1] as well? Otherwise, it'll be reverted the next time closure-library is pumped.
[1]
rules_closure/closure/library/regenerate.py
Line 238 in 5c10507
builds[build].insert( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
strip_prefix = "rules_python-4b84ad270387a7c439ebdccfd530e2339601ef27", | ||
urls = ["https://github.com/bazelbuild/rules_python/archive/4b84ad270387a7c439ebdccfd530e2339601ef27.tar.gz"], | ||
) | ||
|
||
def zlib(): | ||
http_archive( | ||
name = "zlib", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you replace this with build_file = "@com_google_protobuf//:third_party/zlib.BUILD"
and delete the local copy of this file? The one from Protobuf has the necessary load statements for rules_cc
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
@@ -69,6 +69,8 @@ def closure_repositories( | |||
omit_org_ow2_asm_tree = False, | |||
omit_org_ow2_asm_util = False, | |||
omit_phantomjs = False, | |||
omit_rules_proto = False, | |||
omit_rules_python = False, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please also include rules_{cc,java}
. They are required by the native rules, so they are implicit dependencies, but I'm not sure how long they'll stay that way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would rather not add them in this PR, but prefer to add the later. In Gerrit we have also consumed Java rules from Starlark, and not fetched java rules in WORKSPACE file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I think they'll be available at least during 1.x
, so that shouldn't be an issue for the next 3+ month.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, I think rules_{cc,java}
is only available in Bazel 0.29+. Please add them in this PR.
With Bazel 0.28.1, I'm seeing the following error:
ERROR: error loading package 'closure/templates': Unable to find package for @rules_java//java:defs.bzl: The repository '@rules_java' could not be resolved.
INFO: Elapsed time: 0.211s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (3 packages loaded)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added rules_{cc,java}
. Done.
@@ -0,0 +1,13 @@ | |||
java_library( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this added?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
@@ -0,0 +1,23 @@ | |||
// Copyright 2016 The Closure Rules Authors. All Rights Reserved. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this added (dito *.java~
)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM % test failures
@@ -69,6 +69,8 @@ def closure_repositories( | |||
omit_org_ow2_asm_tree = False, | |||
omit_org_ow2_asm_util = False, | |||
omit_phantomjs = False, | |||
omit_rules_proto = False, | |||
omit_rules_python = False, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I think they'll be available at least during 1.x
, so that shouldn't be an issue for the next 3+ month.
6374ae4
to
57e7ced
Compare
@Yannic @gkdn @dslomov @laurentlb I rebased this PR on recent master and squashed all commits. PTAL. |
OK, there is one complication. I conducted custom
Bazel version is Bazel@HEAD, built from this commit (6470bb74047a76d3e0271273ac0864b697ab2aa9). [1] https://github.com/davido/rules_closure/releases/tag/v0.25 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Travis is going to need a bit of extra love because it's using Bazel 0.22.0
.
Otherwise, LGTM
@davido Ok, I was able to reproduce that locally with Gerrit. I think it's caused by the fact that we tell users to use The reason this doesn't fail on our CI is that we use Fixing this is going to be a bit of a pain because it's an incompatible change.
|
Can this now be merged, or do we waiting for something? |
@davido Can you rebase to resolve the merge conflict and update Travis to @gkdn @dslomov Please cut a new release before merging this. There is a chance that this breaks users if they don't explicitly declare |
9c4d754
to
cda1697
Compare
Done. |
@Yannic The build on Mac OS slave is failing:
I remember we had a similar issue in gerrit and solution was to add the following options in
But note, that problem occurred on Gerrit@Bazel CI (Buildkite), so I am not sure about Travis CI, that is used here. //CC @brandjon |
cda1697
to
12dee0f
Compare
Yeah, Bazel 0.25 changed the default version of Python, which I think is why we're seeing this. I'd rather not set any |
12dee0f
to
c4bb4b9
Compare
Ah, OK. I removed |
c4bb4b9
to
4ecd9c1
Compare
@davido's error log indicates that your build is trying to run a py_binary (depswriter) in the host config. By default, the host config uses PY3 (regardless of python_version). This fails at execution time because your machine apparently does not have a python3 interpreter. So presumably, you don't want to be using Python 3 in your build. You can change the default version used by the host config to PY2 by passing --host_force_python=PY2 on the command line. If for some reason you can't do that -- for instance, if you have targets that depend on Bazel thinking that they're PY3 (in |
@brandjon Thanks for your comment. Even after adding I remember one issue in Gerrit where |
Sorry, my bad. The correct way to declare Python 3 is If this also doesn't work, |
4ecd9c1
to
05b429a
Compare
OK, changed. |
Sorry for the repeated changes, fixing Travis really isn't fun. Please also remove |
05b429a
to
4b6723c
Compare
Er, my previous answer was predicated on the assumption that this target is written for PY2 and will only run under PY2. If you actually want it to run under PY3, then presumably the CI config just needs to ensure Python 3 is available. Note that |
4b6723c
to
55cd0bc
Compare
Ok, we're at the point where I'm out of ideas how to make Travis happy. If you want to remove Thank you for your patience! |
55cd0bc
to
78ce05b
Compare
Done. |
closure/repositories.bzl
Outdated
urls = [ | ||
"https://mirror.bazel.build/github.com/google/protobuf/archive/v3.9.0.tar.gz", | ||
"https://github.com/protocolbuffers/protobuf/archive/v3.9.0.tar.gz", | ||
# "https://mirror.bazel.build/github.com/google/protobuf/archive/v3.10.0-rc1.tar.gz", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The commented out mirror URL can be improved here and in other place.
@meteorcloudy @laurentlb Can you please upload https://github.com/protocolbuffers/protobuf/archive/v3.10.0-rc1.tar.gz
to https://mirror.bazel.build/github.com/google/protobuf/archive/v3.10.0-rc1.tar.gz
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it seems like 3.10.0
was just released yesterday. maybe the -rc1
suffix can be dropped?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thank you for working on this btw!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for letting me know, I will amend this series and bump right to 3.10.0.
@meteorcloudy @laurentlb Could you please upload https://github.com/protocolbuffers/protobuf/archive/v3.10.0.tar.gz to bazel mirror? Thank you in advance!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks to everyone for help!
@gkdn @laurentlb @dslomov PTAL, it's ready now.
a6f9801
to
589fe8d
Compare
closure/library/BUILD
Outdated
package(default_visibility = ["//visibility:public"]) | ||
|
||
licenses(["notice"]) | ||
|
||
load("//closure:defs.bzl", "closure_css_library") | ||
load("//closure:defs.bzl", "closure_js_library") | ||
load("//closure:defs.bzl", "closure_css_library", "closure_js_library") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pls move next to other load
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
@@ -12,6 +12,8 @@ | |||
# See the License for the specific language governing permissions and | |||
# limitations under the License. | |||
|
|||
load("@rules_proto//proto:defs.bzl", "ProtoInfo") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pls keep loads together. and for others as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Here and in all other places. To my defense, I should mention, that the changes were done by buildifier and I haven't edited/moved the new introduced load statements to group them together with the existing ones.
|
||
os: | ||
- linux | ||
- osx |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no more osx? I don't have the context.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are these statements in the commit message not enough?
This change also does the following:
[...]
* Bump Bazel version to 0.29.1 in .travis.yml file
[...]
Travis changes:
* Remove dist: trusty (Ubuntu 14.04 is EOL, default is 16.04)
* Remove osx OS as it currently doesn't support Python: [1].
[1] https://github.com/travis-ci/travis-ci/issues/2312
As pointed out by @Yannic OSX is verified in Buildkite pipeline already.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, it's verified by Buildkite (and Travis is super slow), so there's no harm in removing it.
See also #422 (comment)
589fe8d
to
bd812a4
Compare
This change also does the following: * Add fixes for --incompatible_load_{java|proto|python}_rules_from_bzl * Consume zlib.BUILD from protobuf repository * regenerate.py: load py_binary from rules_python * Add rules_cc and rules_java * Bump Bazel version to 0.29.1 in .travis.yml file Travis changes: * Remove dist: trusty (Ubuntu 14.04 is EOL, default is 16.04) * Remove osx OS as it currently doesn't support Python: [1]. [1] travis-ci/travis-ci#2312
@laurentlb Can you please approve this PR? |
I merged the patch. I also cut a new release before merging. |
Thanks! |
Thanks! I'll send a follow-up of #423 to update the readme / getting started section tomorrow. |
The latest version updates protobuf version to 3.10.0: [1]. [1] bazelbuild/rules_closure#413 Change-Id: Ib0aecd23a95f8834f3d06edd66af76a475d0ebb5
This change also does the following: * Add fixes for --incompatible_load_{java|proto|python}_rules_from_bzl * Consume zlib.BUILD from protobuf repository * regenerate.py: load py_binary from rules_python * Add rules_cc and rules_java * Bump Bazel version to 0.29.1 in .travis.yml file Travis changes: * Remove dist: trusty (Ubuntu 14.04 is EOL, default is 16.04) * Remove osx OS as it currently doesn't support Python: [1]. [1] travis-ci/travis-ci#2312
This change also does the following: * Add fixes for --incompatible_load_{java|proto|python}_rules_from_bzl * Consume zlib.BUILD from protobuf repository * regenerate.py: load py_binary from rules_python * Add rules_cc and rules_java * Bump Bazel version to 0.29.1 in .travis.yml file Travis changes: * Remove dist: trusty (Ubuntu 14.04 is EOL, default is 16.04) * Remove osx OS as it currently doesn't support Python: [1]. [1] travis-ci/travis-ci#2312
The JS tests are failing:
That is defined here: [1] and that was added here: [2]. It seems that closure compiler has to be bumped as well.
[1] https://github.com/google/closure-library/blame/master/closure/goog/crypt/base64.js#L71
[2] https://github.com/google/closure-library/releases/tag/v20190729