Often when we do many table_diffs, we want to store the results in some dataframe or tabular format (e.g. in a gsheet somewhere) to do follow-up analysis and diagnosis/resolution work.
This is a quote from Fivetran as one of our customers.
This is actually really meaningful to see diffs between data diff iterations as someone is developing.
I recommend an option like this that creates new files on each invocation similar to how sqlmesh creates log files today OR the option to append to the same file and add timestamps for when diff results were inserted.
sqlmesh table_diff prod:dev -m '*' --store-results append: saves all the results in a clean, ergonomic table format for information from the cli output. We we can store the join on condition, table names, schema diff, sample diff within a single csv and appends to an existing file. This can be the default.
sqlmesh table_diff prod:dev -m '*' --store-results overwrite: saves all the results in a clean, ergonomic table format for information from the cli output. We we can store the join on condition, table names, schema diff, sample diff within a single csv and overwrites an existing file.
cc @themisvaltinos since you've been working on this.
This is a quote from Fivetran as one of our customers.
This is actually really meaningful to see diffs between data diff iterations as someone is developing.
I recommend an option like this that creates new files on each invocation similar to how sqlmesh creates log files today OR the option to append to the same file and add timestamps for when diff results were inserted.
sqlmesh table_diff prod:dev -m '*' --store-results append: saves all the results in a clean, ergonomic table format for information from the cli output. We we can store the join on condition, table names, schema diff, sample diff within a single csv and appends to an existing file. This can be the default.sqlmesh table_diff prod:dev -m '*' --store-results overwrite: saves all the results in a clean, ergonomic table format for information from the cli output. We we can store the join on condition, table names, schema diff, sample diff within a single csv and overwrites an existing file.cc @themisvaltinos since you've been working on this.