Ability to the write Metadata JSON

[Fokko]

Feature Request / Improvement

Migrated from apache/iceberg#7257

[HonahX]

Hi @Fokko. Is there an update on this issue? I am interested in taking this if it's still open.

In terms of implementation, I was thinking of something like this:

def update_table_metadata(base_metadata: TableMetadata, updates: Tuple[TableUpdate, ...]) -> TableMetadata:
    builder = TableMetadataUpdateBuilder(base_metadata)
    for update in updates:
        builder.update_table_metadata(update)
    return builder.build()

Does this approach align with your expectations?
Thanks!

[Fokko]

@HonahX Yes, I think that is how it should be done (Except the builder pattern, that's very much Java style :). I think we can split the work into several work packages:

• Ability to write the JSON to the object store (that was the intent of this PR).
• Have logic to update the metadata dictionary as you pointed out above. I think we can do this per operation (update schema, update partition-spec, update sort-order, etc) to keep it small and we can get it in quickly.
• Implement the commit method per catalog to update the properties (point to the latest metadata, and set the previous metadata).

[foss-v]

Guys, is this a prerequisite for altering the write-ordered-by on a table?

[HonahX]

Hi @vrd83. It depends on which catalog you want to use to alter the table.

For the RestCatalog, this is not a prerequisite. To enable altering the write order, we can implement a ReplaceSortOrder ( UpdateSortOrder) in a similar manner to how we currently support UpdateSchema. You can see how UpdateSchema is implemented here for reference:

iceberg-python/pyiceberg/table/__init__.py

Lines 1314 to 1319 in 03fa9f0

	class UpdateSchema:
	_table: Table
	_schema: Schema
	_last_column_id: itertools.count[int]
	_identifier_field_names: Set[str]

iceberg-python/pyiceberg/table/__init__.py

Lines 264 to 270 in 03fa9f0

	def update_schema(self) -> UpdateSchema:
	"""Create a new UpdateSchema to alter the columns of this table.
	Returns:
	A new UpdateSchema.
	"""
	return UpdateSchema(self._table, self)

The rest server, such as https://github.com/tabular-io/iceberg-rest-image, will handle the metadata update and commit.

For other catalogs, such as Glue, DynamoDB, SQL, and Hive, this will be a prerequisite. We need the ReplaceSortOrder thing and the three bullet points listed by @Fokko above.

@Fokko, please correct me if I've missed anything about the RestCatalog part

[HonahX]

@Fokko Thanks for the explanation!

Ability to write the JSON to the object store (that was the intent of this PR).

I think we already support this:

iceberg-python/pyiceberg/catalog/__init__.py

Lines 571 to 572 in 8e8d39d

	def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) -> None:
	ToOutputFile.table_metadata(metadata, io.new_output(metadata_path))

Could you please elaborate on this a bit? Are there additional features we are looking to implement?

Have logic to update the metadata dictionary as you pointed out above. I think we can do this per operation (update schema, update partition-spec, update sort-order, etc) to keep it small and we can get it in quickly.

I plan to start with update_schema, set_snapshot_ref, and add_snapshot, given that update_schema is already supported and the other two operations are pivotal for write support.

I will try to make a draft PR soon for further discussion.

Thanks!

[Fokko]

@Fokko, please correct me if I've missed anything about the RestCatalog part

No I think that's about right.

Could you please elaborate on this a bit? Are there additional features we are looking to implement?

Ah, I see. You're right there. I was thinking about the logic of generating a filename and doing the actual write.

I plan to start with update_schema, set_snapshot_ref, and add_snapshot, given that update_schema is already supported and the other two operations are pivotal for write support.

That sounds like a great start to me 👍