msg_16d3f84824cd worker_done from structworker_ef74c6dd-9d47-41ce-a55d-f574cf76de40
From mnema-talk v3.3.13, rendered on this site.
PR #771 review: no blocker, 1 should-fix (truncated picture part accepted), 6 notes; S1-S3, N1, N3, invariants, fuzz verified I reviewed PR #771 at head db9576210 against origin/main 77f79b051 read-only, with my own inputs built in a scratch worktree (now removed; nothing edited, committed or pushed). S1, S2, S3, N1, N3 and all standing invariants are verified; there is no blocker, one should-fix (a file part without closing boundary is now accepted for pictures) and six notes. What is left: the author fixes F1, the coordinator decides on the notes.
FINDINGS
F1 should-fix - internal/imgstrip/imgstrip.go:138-144 (take), jpeg.go:80-82, webp.go:77-79, effect in internal/media/stream.go:118-131. The filter maps io.ErrUnexpectedEOF to io.EOF, but that is also the error the SOURCE reports for an upload that stops early (multipart.Part.Read without closing boundary). Input: form body "--boundary\r\nContent-Disposition: form-data; name="file"; filename="a.bin"\r\nContent-Type: image/png\r\n\r\n" + first 2/3 of a PNG, no closing boundary, via Handler.readUpload with a MemoryStore: PNG and JPEG give err=nil, object stored (109 of 164 bytes, 968 of 1452); the same form with GIF or text/plain gives 400 "invalid multipart form" and no object, which is what main does for every type (the next NextPart returns "multipart: NextPart: EOF", which readParts takes for the end of the form, so only Store.Upload's read error rejected it before). At imgstrip level: a reader that ends with io.ErrUnexpectedEOF returns err=nil for JPEG, PNG, WebP buffered and WebP streamed; any other error is passed on correctly in all paths. Limit of the damage: a connection that really breaks (HTTP/1.1 with Content-Length, client closes early) is still rejected with 400 and nothing stored for all types (tested over a real TCP connection); webhook uploads use the same storePart. Fix: keep the source's own error apart from ReadFull's, e.g. wrap src in a small reader that records any error other than io.EOF and let take/jpegData/webpRead turn ErrUnexpectedEOF into EOF only when that record is empty; add the PNG/JPEG/WebP cases to TestStreamFormMapsTransportAndFramingFailures.
F2 note - webp.go:27 and 148-149, webp_stream.go:53. Chunks inside ANMF frames are not looked at in either path. Input: VP8X(0x0E) + ANIM + ANMF{frame header, VP8L, "GPSx", "EXIF", "XMP " each with a marker} + ANMF + top-level EXIF(3) + "XMP ": top-level metadata goes (orientation 3 kept, flag right, idempotent), all three markers inside the frame survive, buffered and streamed. SECURITY.md says so ("the frames of an animated WebP with any chunk inside them"), so the docs are truthful; the spec allows unknown chunks there and decoders skip them, no real writer puts metadata there. Possible fix: walk the ANMF payload after its 16 header bytes and overwrite non VP8/VP8L/ALPH sub-chunks with zeroed JUNK in both paths (keeps all lengths). Same class, also covered by "the kept parts are not examined": ANIM with bytes after its 6, a second VP8L or VP8 chunk holding text: the marker survives.
F3 note - jpeg.go:55-57 and 125-126. The JPEG allow-list covers APPn and COM only. A segment under DNL (FFDC), FFDE, FFDF, JPGn (FFF0-FFFD), reserved FF02-FFBF or FFC8 with a marker string placed after SOI survives whole. Go's decoder rejects these files and libjpeg aborts on all but DNL, cameras do not write them, and the docs say "application segment", so this is truthful; mention it under "what it does not do" or drop segments that are not SOF/DHT/DQT/DRI/SOS/DAC/DNL/EXP/DHP.
F4 note - webp.go:74. Every WebP above 64 KiB allocates limit+1 = 5 MiB whatever its size (measured: 100 KiB file 5,324,816 bytes, 60 KiB file 73,744 bytes). Correct against N1, but N parallel WebP uploads hold N x 5 MiB.
F5 note - imgstrip.go:184-190. Only the first EXIF record counts, also when it is empty or unreadable: PNG with eXIf(0 bytes) then eXIf(orientation 6) ends without orientation. Same rule in JPEG and WebP. Fine as a rule; not in the docs.
F6 note - png.go:50. An unknown CRITICAL chunk (tried XMPD, IDAX, ABCD, JDAT, also IDAt/idat/iend) is dropped like any other and the result is the plain picture. A strict decoder would have refused the source and now shows it; Go's decoder ignored it before as well. I think this is right for a privacy filter (nothing survives, nothing breaks) and matches "any chunk under an unknown name".
F7 note, from reading only, not run - webp_stream.go:50. In the streamed path a VP8X that is not the first chunk keeps its XMP and EXIF flags (the buffered path fixes the flags of the first KEPT chunk). Such a file is invalid anyway.
VERIFIED (own inputs, test files compiled into the packages)
S1 PNG allow-list: verified. Base picture (image/png output: IHDR, IDAT, IEND) plus one chunk carrying a marker, placed after IHDR and after the last IDAT: caBX prVW mkBS zxIf tXMP gIFt gIFx dSIG vpAg meTa iDOT orNT prVt tEXT text TEXt teXt itXt exIf tEXt zTXt iTXt tIME EXIF eXIF EXIf exif sTER oFFs pCAL sCAL gIFg fRAc acTl FCTL fdAt TRNS ICCP iccp plte trns, four zero bytes, four spaces and the critical ones above: result byte-identical to the base picture, CRCs valid. Each of tRNS cHRM gAMA iCCP sBIT sRGB cICP mDCV cLLI bKGD hIST pHYs sPLT acTL fcTL fdAT PLTE passes byte-identical. eXIf: orientation 1-8, II and MM, before and after IDAT, with a second eXIf of another orientation: one eXIf holding exactly the 26-byte record (none for 1), marker gone, first record wins, a 22-byte record and a 380 KiB record work, never larger, idempotent. APNG (IHDR acTL fcTL IDAT fcTL fdAT IEND with tEXt, tIME, eXIf, iTXt, zTXt, caBX in between): result byte-identical to the same file built without those chunks, CRCs valid, acTL/fcTL/fdAT in place. Data after IEND is gone.
S2 WebP allow-list: verified with the VP8L, VP8 and ALPH+VP8 bitstreams of three x/image fixtures. Input VP8X(ICC|EXIF|XMP flags) ICCP(odd length) image EXIF(6 + marker) "XMP "(odd) GPSx exif "xmp " "XMP\0" C2PA JUMB junk Exif "vp8 " iccp anmf RIFF LIST, a zero name, a second EXIF, with odd and even sizes, plus data behind the container. Buffered: byte-identical to VP8X(ICC|EXIF) ICCP image EXIF(26-byte record), so RIFF length, flags and padding are right. Streamed (limits 0, 11, 12, 30, len-1): byte-identical to the same file with XMP flag cleared, the first EXIF holding record + zeros and every other chunk as JUNK of the same length with zero bytes. Streaming forced by size (6 MiB "XMP " chunk, limit 5 MiB): same length, no marker, strict parse passes, JUNK chunks all zero, orientation 6 with flag. Pixels equal in all cases. Sub-chunks inside ANMF: not handled, see F2.
S3: verified. VP8X flags 0x00 + EXIF orientation 1-8 (II, MM, with and without the "Exif\0\0" prefix), buffered: orientation kept, EXIF flag set (no record and no flag for 1). Same file streamed: EXIF becomes JUNK, flag stays clear, orientation lost. Without VP8X: buffered gives the bare image chunk, streamed gives image + JUNK, orientation lost. Announced EXIF streamed: kept; chunks of 0-9 bytes become zero bytes, 10-21 an empty TIFF, 22 and more the orientation record. All of this is what SECURITY.md and doc/upgrade.md say.
N1: verified with runtime.MemStats TotalAlloc. Limit 5 MiB: file of exactly 5 MiB allocates 5,324,928 bytes (1.02x, 4 allocations), 5 MiB - 1 the same, 5 MiB + 1 streams with 5,443,864 bytes in total (the read-ahead buffer once). Not 3x.
N3: verified. Marker text, text with 00/01 bytes, FF 00, FF FF FF 01 FF D0 and text ending in FF FF placed between every pair of segments before the first SOS: result byte-identical to the plain JPEG. In video-001.progressive.jpeg an EXIF, a COM and an APP13 segment before the DHT between scan 1 and 2 and raw text after that DHT: result byte-identical to the fixture. Segments: JFIF with thumbnail comes out as 14 bytes with a 0x0 thumbnail, ICC kept, Adobe cut to 12 bytes; JFXX, Exif, both XMP forms, MPF, FPXR, APP11 JUMBF, APP13, APP3-APP15, COM and look-alikes (jfif, "JFIF" without the zero, icc_profile, adobe) are gone, as is data after EOI.
Invariants: verified on 218 files (GOROOT image/testdata, image/png/testdata with pngsuite, image/gif/testdata, golang.org/x/image@v0.47.0 testdata, the PR's three fixtures): 32 JPEG, 99 PNG, 18 WebP filtered as their own type (WebP at limits 5 MiB, 0 and len/2), every file also through the two other filters and as image/gif. No panic; never larger; filtering twice changes nothing; same result when the source is read byte by byte, through a half/data+error reader and when the result is read 3 bytes at a time; a file of another type is byte-identical (GIF, BMP, TIFF and so on, 11 non-pictures including "RIFF....WAVE", a RIFF/WEBP with length 3 and a cut PNG signature); other MIME strings (image/gif, video/mp4, image/apng, image/pjpeg, empty) return the source reader itself; every source that decodes (32/32, 95/99, 18/18) decodes to identical pixels; a marker appended to the file never survives. Every fixture cut at about 400 offsets: no panic, never larger, idempotent, 0 exceptions. Orientation 1-8 written into every decodable fixture and read back with my own parser: 256 JPEG and 760 PNG cases right, WebP right in S3 on three bitstreams (in this run the only VP8X fixture, phone-orientation-6.webp, already has an EXIF, so my added record came second and 6 won, as designed).
Fuzz: FuzzJPEG 429,142 execs, FuzzPNG 165,833, FuzzWebP 77,615, FuzzWebPStream 145,663, 30 s each with 4 workers, all PASS, no crasher.
Docs: SECURITY.md ("Metadata of picture attachments") and doc/upgrade.md describe what the code does: allow-list in all three formats with the same PNG and WebP names as the code (png.go:18-23, webp.go:27), trailing data, JFIF/JFXX thumbnails, motion-photo video and gain map, C2PA, colour space named in EXIF only shown as sRGB, GIF/MP4/PDF/SVG untouched, the 5 MiB WebP limit (stripBufferBytes = MaxAvatarBytes = 5 << 20, boundary measured), JUNK with zero bytes, the three orientation cases of WebP, the 64 KiB search window, the unexamined kept parts, new uploads only, stored size. "By a member, a bot or a webhook" holds: storePart is the only attachment path and webhookfiles.go:23 uses it. Not stated: F3 and F5; F1 contradicts nothing in the docs but changes behaviour silently.
My test files are at <scratch directory, not kept> and media_review_test.go (not durable; drop them into internal/imgstrip and internal/media as *_test.go, with REVIEW_GOROOT and REVIEW_XIMAGE set, to rerun).