SNOW-1830524 Add decoder logic for Dataframe.join
#2802
Conversation
| @@ -6,9 +6,10 @@ df2 = session.create_dataframe([[1, 2, 3, 4, 5]], schema=['\"A\"','\"B\"','\"C\" | |||
|
|
|||
| df3 = df1.filter(col("\"A\"") == 1).join(df2.select((col("\"A\"") + 1).as_("\"A\""), col("\"B\""), col("\"C\""), col("\"l_0001_C\""), col("\"l_0003_B\""))) | |||
|
|
|||
| df4 = df3.sort(df3.columns) | |||
| # Commented out since df3.columns produces different results in the first encoding and in the encode-decode-encode result. | |||
There was a problem hiding this comment.
Can you find out why this is happening? We shouldn't be removing valid test cases here. I don't see a reason why encode-decode-encode needs to be value equivalent, this is a good example, as long as they are semantically equivalent (the uniqueness of generated columns correspond correctly) that is all that is required.
There was a problem hiding this comment.
The unique columns correspond to each other in both cases but the values are different. I'm not sure what the best way around this is since hardcoding the decoder seems like a bad idea. I can add the test back in.
I'm not familiar with how the column names are generated but can try to figure that out.
…wflakedb/snowpark-python into vbudati/SNOW-1830524-decoder-df-join
| @@ -291,6 +291,7 @@ def test_ast(session, tables, test_case): | |||
| decoder = Decoder(session) | |||
| session._ast_batch.reset_id_gen() # Reset the entity ID generator. | |||
| session._ast_batch.flush() # Clear the AST. | |||
| global_counter.reset() | |||
There was a problem hiding this comment.
Don't understand why this is needed ?
There was a problem hiding this comment.
When two tables are joined and a new column needs to be created with a randomly generated name, this generation is seeded for testing purposes and it depends on the global counter -- so in order for the randomness to be deterministic in the same way, the global counter needs to be at the same "count" (which is reset for every test run).
sfc-gh-vbudati
merged commit a7f5670
into
vbudati/SNOW-1794510-merge-decoder
Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-1830524
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
Added decoder logic for
Dataframe.join. All join tests should work now.