Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HIVE-28033: Fix TestMapJoinOperator to run on Tez #5615

Merged
merged 1 commit into from
Jan 28, 2025

Conversation

Indhumathi27
Copy link
Contributor

@Indhumathi27 Indhumathi27 commented Jan 17, 2025

What changes were proposed in this pull request?

The configuration for HIVE_DEFAULT_NULLS_LAST is set to false when Tez is used. The change ensures compatibility with Tez and proper null handling during serialization.

Why are the changes needed?

With MapReduce (MR), SERIALIZATION_NULL_SORT_ORDER is not set, hence the default null sorting is NULL FIRST. For Tez, the null sorting needed to be adjusted to prevent EOFException and ensure correct test execution with the generated data.

Does this PR introduce any user-facing change?

No

Is the change a dependency upgrade?

No

How was this patch tested?

Test case

@Aggarwal-Raghav
Copy link
Contributor

LGTM +1 (non-binding)

@difin
Copy link
Contributor

difin commented Jan 20, 2025

LGTM, +1

@difin difin self-requested a review January 20, 2025 22:06
@Indhumathi27
Copy link
Contributor Author

@difin can you merge if the PR is okay

@Indhumathi27
Copy link
Contributor Author

@okumin Can you help with review and merging this PR

@okumin
Copy link
Contributor

okumin commented Jan 28, 2025

I confirmed this part is applied only to Tez and that's why.

if (HiveConf.getVar(conf, ConfVars.HIVE_EXECUTION_ENGINE).equals("tez")) {
// In tez we use a different way of transmitting the hash table.
// We basically use ReduceSinkOperators and set the transfer to
// be broadcast (instead of partitioned). As a consequence we use
// a different SerDe than in the MR mapjoin case.
StringBuilder order = new StringBuilder();
StringBuilder nullOrder = new StringBuilder();
for (FieldSchema f: fieldSchemas) {
order.append("+");
nullOrder.append(NullOrdering.defaultNullOrder(conf).getSign());
}

Copy link
Contributor

@okumin okumin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@okumin okumin merged commit 36608b4 into apache:master Jan 28, 2025
6 checks passed
@Indhumathi27
Copy link
Contributor Author

Thanks @okumin @Aggarwal-Raghav @difin for the review

henrib pushed a commit to henrib/hive that referenced this pull request Jan 31, 2025
…umathi Muthumurugesh, reviewed by Raghav Aggarwal, Dmitriy Fingerman, Shohei Okumiya)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants