[format][python] Add packed video frame storage - #9461
Conversation
| # retain count-only sentinels and let VideoFileMeta resolve them. | ||
| self.blob_lengths = [0] * self._video_meta.record_count | ||
| self.blob_offsets = [0] * self._video_meta.record_count | ||
| return |
There was a problem hiding this comment.
We can expose record_count directly from _video_meta instead of allocating these two O(frame count) lists?
There was a problem hiding this comment.
Fixed in 1c0708d. FormatBlobReader now exposes record_count directly from VideoFileMeta for .video files, BlobFallbackBatchReader uses it, and the two O(frame count) sentinel lists are no longer allocated. Added a regression test for selected video rows.
| return; | ||
| } | ||
| ordinal = physicalVideoLengths.size(); | ||
| physicalVideoLengths.add(length); |
There was a problem hiding this comment.
PayloadWriter.write() may return zero, but VideoFileMeta rejects non-positive lengths. Python reject empty payloads here.
There was a problem hiding this comment.
Fixed in 99060ef. RawVideoPayloadWriter now rejects zero-length encoded video payloads, matching the Python writer and VideoFileMeta contract. Added a Java regression test.
| self._ensure_process_local_cache() | ||
| single_row = isinstance(rows, Mapping) | ||
| input_rows = [rows] if single_row else list(rows) | ||
| decoded_rows = [self._decode_row(row) for row in input_rows] |
There was a problem hiding this comment.
Nit: VideoFrameCollator adds ~4.6% sequential decode overhead on droid_100, while direct .video range decoding matches raw MP4. video? We can create. a batch fast path PR as follow-up.
There was a problem hiding this comment.
Agreed. I kept the current collator API unchanged and will leave batch/range decode optimization to a focused follow-up PR so this format PR stays scoped.
|
+1 |
2 participants
Purpose
Add a self-contained video file format for append/data-evolution tables whose logical rows represent frames while the physical decoding unit is a complete encoded video.
This avoids storing one MP4 payload per frame, avoids a sidecar object per video, and leaves the existing
.blobformat unchanged.Design
video-frame-fieldtable option for one scalar BLOB column..videofiles with:VideoFrameDescriptor, extending the normal BLOB descriptor with a presentation-order frame ordinal..videopacks self-contained. They never reference payloads owned by another Paimon data file.PyPaimon
add_videoandadd_videosfor complete-video ingestion.replace_videofor updating the video backing rows selected by a predicate..videofile.update()assignments to the video field and direct callers toreplace_video().blob-as-descriptor=false.VideoFrameCollatorfor PyTorch DataLoader workers with a process-local bounded decoder cache.Compatibility
.blobformat and its readers/writers are unchanged..video.Tests
git diff --checkpassed.torchdependency is not installed; the decoder/collator unit tests pass.