Inspect static PDF and OOXML content with bounded local coverage
This commit is contained in:
Karl committed
1 parent
69d820e315
commit
9e7cb57093
14 files changed
+758
-51
No files matched your search
@@ -4,7 +4,8 @@ All inspection is local. No scan downloads models or calls cloud inference.
|
||||
|
||||
- YARA: genuine pinned yara-python 4.5.4 in private Windows CPython 3.13 runtime. Centrally supplied sources have bounded size and count, includes disabled, imported modules disallowed, compile warnings treated as errors, and five-second match timeout. Rules compile from in-memory strings and scan bytes. Errors, unavailable engine, missing rules and oversize payloads block downloads. Compilation currently runs inside the inspector process; a native compiler hang is bounded by the browser's inspector request deadline but does not restart the inspector automatically.
|
||||
- Text: UTF-8 extraction and deterministic secret/literal/fingerprint classification.
|
||||
- PDF/OOXML: bounded extraction of text and embedded images; full rendered visual coverage is incomplete and outbound DLP fails closed.
|
||||
- PDF: bounded static subset now inspects decoded text, metadata and all indexed objects, independently OCRs embedded page images, and renders/OCRs every page with the bundled offline PDFium/OCR runtime. Active content, forms, annotations, attachments, layers, incremental updates, orphan/unsupported streams and embedded/custom font encodings remain blocked. See [PDF coverage](PDF-COVERAGE.md).
|
||||
- OOXML: bounded extraction of text and embedded images; full rendered visual coverage is incomplete and outbound DLP fails closed.
|
||||
- Images: bundled RapidOCR 3.8.1 with CPU ONNXRuntime 1.24.4 and packaged PP-OCRv4 detection/recognition plus v2 classification weights. Explicit verified local model paths bypass all model downloads. Real Windows test renders a secret into an image and blocks it with network model downloading forbidden. Models primarily support Chinese/English; other languages and recognition accuracy require validation. OCR completion indicates all frames processed, not guaranteed recognition accuracy. Pillow parsing has dimension/frame limits. Optional `SAFETYNET_OCR_ENGINE=tesseract` uses administrator-installed trusted `TESSERACT_PATH`, optionally `TESSDATA_PREFIX` for local traineddata. Tesseract weights are not bundled. Missing OCR blocks DLP.
|
||||
- Audio/video: recognition of container types only. Transcription and full video inspection are unconfigured; DLP blocks rather than claiming safe coverage.
|
||||
- Semantic adapter: real Google EmbeddingGemma 2, Q8_0 GGUF export revision `bfcd298762cc34d0357ece5ebdd31791a3a374d8`, 768 dimensions. A verified native worker runs explicitly on CPU, reads bounded local NDJSON, and returns actual normalized vectors. Both GGUF weights and native runtime modules must pass size/SHA-256 checks before loading. Configure `EMBEDDING_MODEL_DIR` and `EMBEDDING_WORKER_PATH` to staged, protected local resources. No cloud inference, model download or Python ML framework is used. Text uses the same pinned `document-v1` profile for training and inspection; native token windows must cover exact UTF-8 bytes through the final byte, otherwise scanning fails closed. At most 64 vectors are allowed; incomplete or oversized training is refused. Native image embeddings process all bounded frames resized to at most672px with48px alignment. Native audio supports only mono16kHz PCM16 WAV up to30seconds; other media remains uninspectable. Actual model fixtures must run separately from protocol unit tests to establish operational readiness.
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
# Local static PDF coverage
|
||||
|
||||
The inspector uses hash-pinned `pypdfium2 5.14.0` with official Windows x64 PDFium `156.0.8076.0`. The wheel has no V8 or XFA feature flags. No JavaScript, forms, external URLs or network rendering are enabled. This does not modify Chromium or Electron.
|
||||
|
||||
For the supported subset, inspection combines decoded page text, every indexed object's strings and metadata, decoded content operands, independent OCR of original page image resources, and rendering/OCR of every page. Rendered pages and original images are also supplied to local semantic analysis when policy requires it. Rendering alone is never treated as coverage of hidden bytes.
|
||||
|
||||
Limits include 32 pages, 10,000 objects/operations, depth 40, 20 million pixels per rendered page, 50 million rendered pixels total, 20 MiB decoded streams/media, 64 media items, one million extracted characters and a 20-second cooperative PDF deadline. The existing isolated worker enforces an external deadline and remains in the inspector's 4 GiB Windows Job Object. Decoder output, declared lengths, filter chains and decoding work are bounded before decompression via the installed pypdf configuration API. The external process deadline remains necessary for native rendering/OCR calls.
|
||||
|
||||
The supported font subset is the 14 standard PDF Type 1 fonts with standard, WinAnsi or MacRoman encoding. Embedded fonts, custom encoding/CMaps, active actions, annotations, forms, optional layers, attachments, incremental revisions, hidden/orphan streams and unsupported resources fail closed. This is **not full compatibility with common PDF exports**, which often embed fonts or use XMP metadata. Broader support needs independent coverage tests rather than changing the completeness flag.
|
||||
|
||||
`coverage.complete` means every component in this explicit processing subset was inspected; OCR and classification remain probabilistic and can miss visually ambiguous information. It does not certify that a document is harmless.
|
||||
|
||||
Actual offline tests cover benign PDF release, hex-encoded text secrets, metadata and unreferenced strings, vector-only rendered lettering without extractable text, embedded raster secrets, active/encrypted/hidden content rejection and a compressed expansion bomb rejected before OCR. These tests use the real renderer and local OCR, apart from the explicit missing-OCR failure test.
|
||||
|
||||
PDF timing stages are fixed `pdf.structure`, `pdf.images`, `pdf.render` and `pdf.ocr`, under the existing private `parse`/`inspect` stages. They contain timings and correlation, never document text.
|
||||
|
||||
Sources: [pypdfium2 release and licensing](https://pypi.org/project/pypdfium2/5.14.0/), [official PDFium API and process-safety guidance](https://pypdfium2.readthedocs.io/en/stable/python_api.html). The runtime retains the wheel's BSD/Apache and PDFium dependency license notices in its distribution metadata.
|
||||
@@ -200,51 +200,12 @@ def extract(data, kind):
|
||||
text,complete=ocr_image(data)
|
||||
return text,complete,[("image",data)]
|
||||
if kind=="pdf":
|
||||
from pypdf import PdfReader
|
||||
reader=PdfReader(io.BytesIO(data),strict=True)
|
||||
if reader.is_encrypted: raise ValueError("Encrypted PDF cannot be inspected")
|
||||
if len(reader.pages)>100: raise ValueError("PDF page budget exceeded")
|
||||
text=[];complete=True;media=[]
|
||||
for page in reader.pages:
|
||||
value=page.extract_text() or ""
|
||||
text.append(value)
|
||||
images=list(page.images)
|
||||
if not value.strip() and not images: complete=False
|
||||
for image in images:
|
||||
if len(image.data)>MAX_BYTES: raise ValueError("PDF image budget exceeded")
|
||||
scanned,ok=ocr_image(image.data);text.append(scanned);complete=complete and ok
|
||||
media.append(("image",image.data))
|
||||
if len(media)>100: raise ValueError("PDF media budget exceeded")
|
||||
# Attachments, JavaScript, forms, actions and vector drawings need deeper inspection.
|
||||
if page.get("/Annots"): complete=False
|
||||
root=reader.trailer["/Root"]
|
||||
if root.get("/AcroForm") or root.get("/OpenAction") or root.get("/Names"): complete=False
|
||||
# Text extraction is useful, but does not prove all visible/vector PDF content was covered.
|
||||
return "\n".join(text),False,media
|
||||
from pdf_documents import extract_pdf
|
||||
return extract_pdf(data, ocr_image)
|
||||
if kind=="office":
|
||||
from defusedxml.ElementTree import fromstring
|
||||
text=[];media=[];complete=True
|
||||
with zipfile.ZipFile(io.BytesIO(data)) as archive:
|
||||
members=archive.infolist()
|
||||
if len(members)>2000 or sum(x.file_size for x in members)>MAX_BYTES:
|
||||
raise ValueError("Archive expansion budget exceeded")
|
||||
if not any(x.filename in ["word/document.xml","ppt/presentation.xml","xl/workbook.xml"] for x in members):
|
||||
raise ValueError("Only OOXML document archives are supported")
|
||||
for entry in members:
|
||||
if entry.flag_bits&1 or entry.file_size>MAX_BYTES or entry.file_size>max(entry.compress_size,1)*200:
|
||||
raise ValueError("Unsafe document archive")
|
||||
name=entry.filename
|
||||
if any(s in name.lower() for s in ["vbaproject","embeddings/","activex/","customxml/"]): complete=False
|
||||
if name.endswith(".xml"):
|
||||
root=fromstring(archive.read(entry))
|
||||
text.extend(element.text for element in root.iter() if element.tag.rsplit("}",1)[-1] in ["t","v"] and element.text)
|
||||
elif "/media/" in name:
|
||||
payload=archive.read(entry)
|
||||
if content_kind(payload)!="image": complete=False; continue
|
||||
scanned,ok=ocr_image(payload);text.append(scanned);complete=complete and ok;media.append(("image",payload))
|
||||
if len(media)>100: raise ValueError("Office media budget exceeded")
|
||||
# Charts, relationships, equations and rendered layout are not yet fully inspected.
|
||||
return "\n".join(text),False,media
|
||||
from office_content import extract_office
|
||||
result=extract_office(data,ocr_image,max_bytes=MAX_BYTES)
|
||||
return result["text"],result["complete"],result["media"]
|
||||
if kind in ["audio","video"]: return "",False,[(kind,data)]
|
||||
raise ValueError("Unsupported content type")
|
||||
|
||||
|
||||
@@ -0,0 +1,216 @@
|
||||
"""Bounded static OOXML content coverage, not an Office renderer or macro engine.
|
||||
|
||||
OPC and part families: Microsoft Open XML SDK structure documentation:
|
||||
https://learn.microsoft.com/en-us/office/open-xml/general/how-to-create-a-package
|
||||
https://learn.microsoft.com/en-us/office/open-xml/spreadsheet/structure-of-a-spreadsheetml-document
|
||||
https://learn.microsoft.com/en-us/office/open-xml/presentation/structure-of-a-presentationml-document
|
||||
All XML text, tails, attributes, namespace declarations and ZIP metadata are inputs.
|
||||
Only known static XML parts and successfully OCR-inspected raster assets are covered.
|
||||
Unknown parts, equations, vector/custom layouts and active features remain incomplete.
|
||||
"""
|
||||
import io
|
||||
import posixpath
|
||||
import re
|
||||
import struct
|
||||
import unicodedata
|
||||
import zipfile
|
||||
from urllib.parse import urlsplit
|
||||
from defusedxml.ElementTree import fromstring,iterparse
|
||||
from stage_timings import stage
|
||||
CT='http://schemas.openxmlformats.org/package/2006/content-types'
|
||||
REL='http://schemas.openxmlformats.org/package/2006/relationships'
|
||||
R='http://schemas.openxmlformats.org/officeDocument/2006/relationships'
|
||||
W='http://schemas.openxmlformats.org/wordprocessingml/2006/main'
|
||||
S='http://schemas.openxmlformats.org/spreadsheetml/2006/main'
|
||||
P='http://schemas.openxmlformats.org/presentationml/2006/main'
|
||||
A='http://schemas.openxmlformats.org/drawingml/2006/main'
|
||||
CP='http://schemas.openxmlformats.org/package/2006/metadata/core-properties'
|
||||
EP='http://schemas.openxmlformats.org/officeDocument/2006/extended-properties'
|
||||
CUSTOM='http://schemas.openxmlformats.org/officeDocument/2006/custom-properties'
|
||||
TYPE_PREFIX='application/vnd.openxmlformats-officedocument.'
|
||||
MAIN={TYPE_PREFIX+'wordprocessingml.document.main+xml':('docx',W,'document'),TYPE_PREFIX+'spreadsheetml.sheet.main+xml':('xlsx',S,'workbook'),TYPE_PREFIX+'presentationml.presentation.main+xml':('pptx',P,'presentation')}
|
||||
XML_TYPES={TYPE_PREFIX+'wordprocessingml.'+name+'+xml':(W,root) for name,root in [('styles','styles'),('settings','settings'),('fontTable','fonts'),('numbering','numbering'),('header','hdr'),('footer','ftr'),('comments','comments'),('footnotes','footnotes'),('endnotes','endnotes'),('webSettings','webSettings')]}
|
||||
XML_TYPES.update({TYPE_PREFIX+'spreadsheetml.'+name+'+xml':(S,root) for name,root in [('worksheet','worksheet'),('sharedStrings','sst'),('styles','styleSheet'),('table','table')]})
|
||||
XML_TYPES.update({TYPE_PREFIX+'presentationml.'+name+'+xml':(P,root) for name,root in [('slide','sld'),('notesSlide','notes'),('notesMaster','notesMaster'),('slideMaster','sldMaster'),('slideLayout','sldLayout'),('presProps','presentationPr'),('viewProps','viewPr'),('tableStyles','tblStyleLst')]})
|
||||
XML_TYPES['application/vnd.ms-word.stylesWithEffects+xml']=(W,'styles')
|
||||
XML_TYPES[TYPE_PREFIX+'customXmlProperties+xml']=('http://schemas.openxmlformats.org/officeDocument/2006/customXml','datastoreItem')
|
||||
XML_TYPES[TYPE_PREFIX+'presentationml.tableStyles+xml']=(A,'tblStyleLst')
|
||||
XML_TYPES.update({'application/vnd.openxmlformats-package.core-properties+xml':(CP,'coreProperties'),TYPE_PREFIX+'extended-properties+xml':(EP,'Properties'),TYPE_PREFIX+'custom-properties+xml':(CUSTOM,'Properties'),TYPE_PREFIX+'theme+xml':(A,'theme')})
|
||||
SAFE_REL={R+'/'+name for name in ['officeDocument','extended-properties','custom-properties','styles','settings','fontTable','numbering','header','footer','comments','footnotes','endnotes','worksheet','sharedStrings','table','theme','slide','notesSlide','notesMaster','slideMaster','slideLayout','presProps','viewProps','tableStyles','image']}
|
||||
SAFE_REL.update({R+'/customXml',R+'/customXmlProps',R+'/webSettings','http://schemas.microsoft.com/office/2007/relationships/stylesWithEffects','http://schemas.openxmlformats.org/package/2006/relationships/metadata/core-properties','http://schemas.openxmlformats.org/package/2006/relationships/metadata/thumbnail'})
|
||||
SAFE_NS={CT,REL,R,W,S,P,A,CP,EP,CUSTOM,'http://schemas.openxmlformats.org/drawingml/2006/picture','http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing','http://schemas.openxmlformats.org/officeDocument/2006/docPropsVTypes','http://purl.org/dc/elements/1.1/','http://purl.org/dc/terms/','http://purl.org/dc/dcmitype/','http://www.w3.org/2001/XMLSchema-instance','http://www.w3.org/XML/1998/namespace','http://schemas.openxmlformats.org/markup-compatibility/2006','http://schemas.openxmlformats.org/officeDocument/2006/bibliography','http://schemas.openxmlformats.org/officeDocument/2006/customXml','http://schemas.openxmlformats.org/officeDocument/2006/math'}
|
||||
SAFE_EXTENSION_TAGS={'{urn:schemas-microsoft-com:office:office}'+name for name in ['shapedefaults','shapelayout','idmap']}|{'{http://schemas.microsoft.com/office/word/2010/wordml}'+name for name in ['docId','defaultImageDpi']}
|
||||
UNSUPPORTED_TAGS={'AlternateContent','oMath','oMathPara','custGeom','pict','object','control','oleObj','graphicFrame','chart','grpSp','cxnSp','fldSimple','fldChar','instrText','altChunk','externalLink','externalReference','externalReferences','blob','hlinkClick','hlinkHover','animation','transition','timing','embedRegular','embedBold','embedItalic','embedBoldItalic'}
|
||||
class OfficeCoverageError(ValueError):pass
|
||||
|
||||
def _part(name):
|
||||
if not isinstance(name,str) or not name or len(name)>240 or '\\' in name or '%' in name or name.startswith('/') or any(part in {'','.', '..'} for part in name.split('/')) or any(ord(c)<32 for c in name) or unicodedata.normalize('NFC',name)!=name:raise OfficeCoverageError('office-part-name-invalid')
|
||||
return name
|
||||
|
||||
def _physical_zip(data,archive,infos):
|
||||
if not data.startswith(b'PK\x03\x04'):raise OfficeCoverageError('office-archive-prefix-invalid')
|
||||
cursor=0
|
||||
for info in sorted(infos,key=lambda item:item.header_offset):
|
||||
if info.header_offset!=cursor or cursor+30>len(data):raise OfficeCoverageError('office-archive-gap')
|
||||
signature,version,flags,method,time,date,crc,compressed,size,name_length,extra_length=struct.unpack_from('<IHHHHHIIIHH',data,cursor)
|
||||
if signature!=0x04034b50 or flags!=info.flag_bits or method!=info.compress_type:raise OfficeCoverageError('office-local-header-invalid')
|
||||
if extra_length:raise OfficeCoverageError('office-local-extra-uncovered')
|
||||
cursor+=30+name_length+extra_length+info.compress_size
|
||||
if flags&8:
|
||||
if data[cursor:cursor+4]==b'PK\x07\x08':cursor+=4
|
||||
if cursor+12>len(data) or struct.unpack_from('<III',data,cursor)!=(info.CRC,info.compress_size,info.file_size):raise OfficeCoverageError('office-descriptor-invalid')
|
||||
cursor+=12
|
||||
elif (crc,compressed,size)!=(info.CRC,info.compress_size,info.file_size):raise OfficeCoverageError('office-local-size-invalid')
|
||||
if cursor!=archive.start_dir:raise OfficeCoverageError('office-archive-gap')
|
||||
for _info in infos:
|
||||
if data[cursor:cursor+4]!=b'PK\x01\x02' or cursor+46>len(data):raise OfficeCoverageError('office-central-header-invalid')
|
||||
name_length,extra_length,comment_length=struct.unpack_from('<HHH',data,cursor+28);cursor+=46+name_length+extra_length+comment_length
|
||||
if data[cursor:cursor+4]!=b'PK\x05\x06' or cursor+22>len(data):raise OfficeCoverageError('office-end-record-invalid')
|
||||
disk,start_disk,count_disk,count,total_size,start,comment=struct.unpack_from('<HHHHIIH',data,cursor+4)
|
||||
if disk or start_disk or count_disk!=len(infos) or count!=len(infos) or start!=archive.start_dir or total_size!=cursor-archive.start_dir or cursor+22+comment!=len(data):raise OfficeCoverageError('office-end-record-invalid')
|
||||
|
||||
def extract_office(data,ocr_image,*,max_bytes=20*1024*1024,max_parts=2000,max_media=100,max_text=2*1024*1024):
|
||||
if not isinstance(data,bytes) or len(data)>max_bytes or not callable(ocr_image):raise OfficeCoverageError('office-input-invalid')
|
||||
texts=[];seen_text=set();media=[];reasons=set();xml={};images={};types={};members={};payloads={};total=0;nodes=0
|
||||
def append(value):
|
||||
nonlocal total
|
||||
if value:
|
||||
value=str(value)
|
||||
if value in seen_text:return
|
||||
seen_text.add(value);total+=len(value)
|
||||
if total>max_text:raise OfficeCoverageError('office-text-limit')
|
||||
texts.append(value)
|
||||
with stage('office.archive'):
|
||||
with zipfile.ZipFile(io.BytesIO(data)) as archive:
|
||||
infos=archive.infolist()
|
||||
_physical_zip(data,archive,infos)
|
||||
if len(infos)>max_parts or sum(item.file_size for item in infos)>max_bytes:raise OfficeCoverageError('office-expansion-limit')
|
||||
append(archive.comment.decode('utf-8','strict'))
|
||||
for info in infos:
|
||||
name=_part(info.filename)
|
||||
if name.casefold() in members:raise OfficeCoverageError('office-duplicate-part')
|
||||
members[name.casefold()]=(name,info)
|
||||
if info.is_dir() or info.flag_bits&1 or info.compress_type not in {zipfile.ZIP_STORED,zipfile.ZIP_DEFLATED} or info.file_size>max_bytes or info.file_size>max(1,info.compress_size)*200 or (info.external_attr>>16)&0o170000==0o120000:raise OfficeCoverageError('office-archive-unsafe')
|
||||
# Unknown local/central extra fields may hide material outside parts.
|
||||
if info.extra:reasons.add('zip-extra-field-uncovered')
|
||||
append(name);append(info.comment.decode('utf-8','strict'))
|
||||
payload=archive.read(info)
|
||||
if len(payload)!=info.file_size:raise OfficeCoverageError('office-part-size-invalid')
|
||||
payloads[name]=payload
|
||||
with stage('office.xml'):
|
||||
for name,payload in payloads.items():
|
||||
if name.endswith(('.xml','.rels')) or name=='[Content_Types].xml':
|
||||
root=fromstring(payload,forbid_dtd=True,forbid_entities=True,forbid_external=True);xml[name]=root
|
||||
for _event,namespace in iterparse(io.BytesIO(payload),events=['start-ns'],forbid_dtd=True,forbid_entities=True,forbid_external=True):append(namespace[0]);append(namespace[1])
|
||||
for event,item in iterparse(io.BytesIO(payload),events=['comment','pi'],forbid_dtd=True,forbid_entities=True,forbid_external=True):
|
||||
append(item.text)
|
||||
if event=='pi':reasons.add('xml-processing-instruction-unsupported')
|
||||
parents={child:parent for parent in root.iter() for child in parent}
|
||||
append(''.join(value for value in root.itertext() if value.strip()))
|
||||
for element in root.iter():
|
||||
nodes+=1
|
||||
if nodes>100000:raise OfficeCoverageError('office-xml-node-limit')
|
||||
append(element.tag);append(element.text);append(element.tail)
|
||||
for key,value in element.attrib.items():
|
||||
append(key);append(value)
|
||||
if key.startswith('{') and key[1:].split('}',1)[0] not in SAFE_NS and key not in {'{http://schemas.microsoft.com/office/word/2010/wordml}paraId','{http://schemas.microsoft.com/office/word/2010/wordml}textId'} and not (element.tag in SAFE_EXTENSION_TAGS and ((key=='{urn:schemas-microsoft-com:vml}ext' and value=='edit') or (key=='{http://schemas.microsoft.com/office/word/2010/wordml}val' and value.isdecimal()))):reasons.add('xml-namespace-unsupported')
|
||||
namespace=element.tag[1:].split('}',1)[0] if element.tag.startswith('{') else ''
|
||||
if namespace not in SAFE_NS and element.tag not in SAFE_EXTENSION_TAGS:reasons.add('xml-namespace-unsupported')
|
||||
if element.tag.rsplit('}',1)[-1] in UNSUPPORTED_TAGS:reasons.add('xml-feature-unsupported')
|
||||
if element.tag=='{'+A+'}prstGeom' and element.get('prst') not in {'rect','roundRect'}:reasons.add('vector-preset-unsupported')
|
||||
if element.tag=='{'+S+'}f':
|
||||
formula=element.text or '';cached=parents.get(element)
|
||||
cached=cached.find('{'+S+'}v') if cached is not None else None
|
||||
without_strings=re.sub(r'"(?:[^"]|"")*"','',formula)
|
||||
functions={name.upper() for name in re.findall(r'([A-Za-z_][A-Za-z0-9_.]*)\s*\(',without_strings)}
|
||||
safe_functions={'SUM','SUMIF','SUMIFS','COUNT','COUNTIF','COUNTIFS','COUNTA','AVERAGE','AVERAGEIF','AVERAGEIFS','MIN','MAX','ROUND','ROUNDDOWN','ROUNDUP','ABS','IF','AND','OR','NOT'}
|
||||
if not formula or set(element.attrib)-{'ca'} or any(c in without_strings for c in '[]|') or not functions<=safe_functions or cached is None or not (cached.text or '').strip():reasons.add('formula-coverage-incomplete')
|
||||
|
||||
elif payload.startswith((b'\x89PNG\r\n\x1a\n',b'\xff\xd8\xff',b'GIF87a',b'GIF89a')):images[name]=payload
|
||||
else:reasons.add('binary-part-unsupported')
|
||||
with stage('office.relationships'):
|
||||
content=xml.get('[Content_Types].xml')
|
||||
if content is None or content.tag!='{'+CT+'}Types':raise OfficeCoverageError('office-content-types-invalid')
|
||||
defaults={};overrides={}
|
||||
for item in content:
|
||||
if item.tag=='{'+CT+'}Default' and set(item.attrib)=={'Extension','ContentType'}:
|
||||
ext=item.attrib['Extension'].lower()
|
||||
if not re.fullmatch('[a-z0-9]+',ext) or ext in defaults:raise OfficeCoverageError('office-content-types-invalid')
|
||||
defaults[ext]=item.attrib['ContentType']
|
||||
elif item.tag=='{'+CT+'}Override' and set(item.attrib)=={'PartName','ContentType'}:
|
||||
part=item.attrib['PartName']
|
||||
if not part.startswith('/'):raise OfficeCoverageError('office-content-types-invalid')
|
||||
part=_part(part[1:])
|
||||
if part.casefold() in {p.casefold() for p in overrides}:raise OfficeCoverageError('office-content-types-invalid')
|
||||
overrides[part]=item.attrib['ContentType']
|
||||
else:raise OfficeCoverageError('office-content-types-invalid')
|
||||
if any(name.casefold() not in members for name in overrides):raise OfficeCoverageError('office-content-types-dangling')
|
||||
main=[]
|
||||
for name,_info in members.values():
|
||||
if name=='[Content_Types].xml':continue
|
||||
type_=overrides.get(name,defaults.get(name.rsplit('.',1)[-1].lower()))
|
||||
if not type_:raise OfficeCoverageError('office-content-type-missing')
|
||||
types[name]=type_
|
||||
if type_ in MAIN:
|
||||
family,namespace,rootname=MAIN[type_];main.append((name,family))
|
||||
if name not in xml or xml[name].tag!='{'+namespace+'}'+rootname:raise OfficeCoverageError('office-main-part-invalid')
|
||||
elif name.endswith('.rels'):
|
||||
if type_!='application/vnd.openxmlformats-package.relationships+xml':raise OfficeCoverageError('office-relationship-type-invalid')
|
||||
elif type_=='application/xml' and name in xml and xml[name].tag=='{http://schemas.openxmlformats.org/officeDocument/2006/bibliography}Sources':pass
|
||||
elif type_ in XML_TYPES:
|
||||
namespace,rootname=XML_TYPES[type_]
|
||||
if name not in xml or xml[name].tag!='{'+namespace+'}'+rootname:raise OfficeCoverageError('office-xml-part-invalid')
|
||||
elif name in images:
|
||||
expected='image/png' if images[name].startswith(b'\x89PNG') else 'image/jpeg' if images[name].startswith(b'\xff\xd8') else 'image/gif'
|
||||
if type_!=expected:raise OfficeCoverageError('office-image-type-invalid')
|
||||
else:reasons.add('content-type-unsupported')
|
||||
if len(main)!=1:raise OfficeCoverageError('office-main-count-invalid')
|
||||
relationships={};top_main=[]
|
||||
for name,root in xml.items():
|
||||
if not name.endswith('.rels'):continue
|
||||
if root.tag!='{'+REL+'}Relationships':raise OfficeCoverageError('office-relationships-invalid')
|
||||
if name=='_rels/.rels':owner=''
|
||||
elif '/_rels/' in name:
|
||||
prefix,file=name.rsplit('/_rels/',1);owner=prefix+'/'+file[:-5]
|
||||
if owner.casefold() not in members:raise OfficeCoverageError('office-relationships-owner-invalid')
|
||||
else:raise OfficeCoverageError('office-relationships-location-invalid')
|
||||
ids=set()
|
||||
for item in root:
|
||||
if item.tag!='{'+REL+'}Relationship' or not {'Id','Type','Target'}<=set(item.attrib) or set(item.attrib)-{'Id','Type','Target','TargetMode'}:raise OfficeCoverageError('office-relationship-invalid')
|
||||
id_=item.attrib['Id'];target=item.attrib['Target'];type_=item.attrib['Type']
|
||||
if not id_ or id_ in ids:raise OfficeCoverageError('office-relationship-id-invalid')
|
||||
ids.add(id_)
|
||||
if item.attrib.get('TargetMode','Internal')!='Internal':reasons.add('external-relationship-unsupported');continue
|
||||
if urlsplit(target).scheme or '%' in target or '\\' in target or '#' in target or '?' in target:raise OfficeCoverageError('office-relationship-target-invalid')
|
||||
resolved=posixpath.normpath(posixpath.join(posixpath.dirname(owner),target)) if not target.startswith('/') else target[1:]
|
||||
_part(resolved)
|
||||
if resolved.casefold() not in members:raise OfficeCoverageError('office-relationship-target-missing')
|
||||
if type_ not in SAFE_REL:reasons.add('relationship-type-unsupported')
|
||||
else:
|
||||
target_type=types.get(resolved)
|
||||
role=type_.rsplit('/',1)[-1]
|
||||
if role=='officeDocument':valid=target_type in MAIN
|
||||
elif role in {'image','thumbnail'}:valid=resolved in images
|
||||
elif role=='customXml':valid=target_type=='application/xml' and resolved in xml and xml[resolved].tag=='{http://schemas.openxmlformats.org/officeDocument/2006/bibliography}Sources'
|
||||
elif role=='customXmlProps':valid=target_type==TYPE_PREFIX+'customXmlProperties+xml'
|
||||
elif role=='stylesWithEffects':valid=target_type=='application/vnd.ms-word.stylesWithEffects+xml'
|
||||
elif role=='core-properties':valid=target_type=='application/vnd.openxmlformats-package.core-properties+xml'
|
||||
elif role in {'extended-properties','custom-properties'}:valid=target_type==TYPE_PREFIX+role+'+xml'
|
||||
else:valid=target_type in XML_TYPES and (target_type==TYPE_PREFIX+'theme+xml' if role=='theme' else target_type.endswith('.'+role+'+xml'))
|
||||
if not valid:raise OfficeCoverageError('office-relationship-type-target-mismatch')
|
||||
if type_==R+'/officeDocument' and not owner:top_main.append(resolved)
|
||||
relationships[owner]=ids
|
||||
if top_main!=[main[0][0]]:raise OfficeCoverageError('office-root-relationship-invalid')
|
||||
for name,root in xml.items():
|
||||
if name.endswith('.rels') or name=='[Content_Types].xml':continue
|
||||
for element in root.iter():
|
||||
for key,value in element.attrib.items():
|
||||
if key.startswith('{'+R+'}') and value not in relationships.get(name,set()):raise OfficeCoverageError('office-dangling-relationship-reference')
|
||||
with stage('office.ocr'):
|
||||
for name,payload in images.items():
|
||||
if len(media)>=max_media:raise OfficeCoverageError('office-media-limit')
|
||||
scanned,complete=ocr_image(payload)
|
||||
if not isinstance(scanned,str) or complete is not True:reasons.add('image-ocr-incomplete')
|
||||
else:append(scanned)
|
||||
media.append(('image',payload))
|
||||
return {'text':'\n'.join(texts),'complete':not reasons,'media':media,'family':main[0][1],'reasons':sorted(reasons),'parts':len(members)}
|
||||
@@ -0,0 +1,186 @@
|
||||
"""Bounded static PDF inspection: decoded content plus every-page rendering/OCR.
|
||||
|
||||
Active, layered, annotated, embedded-file and incremental documents are outside
|
||||
the supported subset and fail closed. This runs inside the existing isolated
|
||||
parser process; PDFium has no JavaScript/XFA support and forms are never enabled.
|
||||
"""
|
||||
import io
|
||||
import math
|
||||
import time
|
||||
from stage_timings import stage
|
||||
|
||||
MAX_PAGES = 32
|
||||
MAX_OBJECTS = 10000
|
||||
MAX_PIXELS = 50_000_000
|
||||
MAX_PAGE_PIXELS = 20_000_000
|
||||
MAX_BYTES = 20 * 1024 * 1024
|
||||
MAX_TEXT = 1_000_000
|
||||
FORBIDDEN = frozenset(("/Annots", "/AcroForm", "/XFA", "/OpenAction", "/AA", "/A", "/JS", "/JavaScript", "/Names", "/EmbeddedFiles", "/EF", "/AF", "/OCProperties", "/Collection", "/RichMedia", "/Movie", "/Sound", "/3D", "/Rendition", "/Launch", "/URI", "/GoToR", "/GoToE", "/SubmitForm", "/ImportData", "/Encrypt", "/PieceInfo"))
|
||||
STANDARD_FONTS = frozenset(("/Helvetica", "/Helvetica-Bold", "/Helvetica-Oblique", "/Helvetica-BoldOblique", "/Times-Roman", "/Times-Bold", "/Times-Italic", "/Times-BoldItalic", "/Courier", "/Courier-Bold", "/Courier-Oblique", "/Courier-BoldOblique", "/Symbol", "/ZapfDingbats"))
|
||||
|
||||
|
||||
def extract_pdf(data, ocr_image):
|
||||
from pypdf import apply_configuration
|
||||
# Enforce decompressor limits BEFORE any stream get_data()/image allocation.
|
||||
with apply_configuration(maximum_declared_stream_length=MAX_BYTES,
|
||||
array_based_stream_maximum_output_length=MAX_BYTES,
|
||||
zlib_maximum_output_length=MAX_BYTES,
|
||||
lzw_maximum_output_length=MAX_BYTES,
|
||||
run_length_maximum_output_length=MAX_BYTES,
|
||||
brotli_maximum_output_length=MAX_BYTES,
|
||||
jbig2_maximum_output_length=MAX_BYTES,
|
||||
image_maximum_buffer_size=MAX_BYTES,
|
||||
stream_filters_maximum_length=4,
|
||||
stream_decoding_work_maximum_length=MAX_BYTES,
|
||||
zlib_maximum_recovery_input_length=1048576):
|
||||
return _extract_pdf(data, ocr_image)
|
||||
|
||||
|
||||
def _extract_pdf(data, ocr_image):
|
||||
from pypdf import PdfReader
|
||||
from pypdf.generic import IndirectObject, DictionaryObject, ArrayObject, StreamObject, TextStringObject, ByteStringObject, NameObject, ContentStream
|
||||
import pypdfium2 as pdfium
|
||||
if len(data) > MAX_BYTES or data.count(b"%%EOF") != 1 or data.rsplit(b"%%EOF", 1)[-1].strip():
|
||||
raise ValueError("Unsupported PDF container")
|
||||
if pdfium.PDFIUM_INFO.flags or pdfium.PYPDFIUM_INFO.api_tag != (5, 14, 0):
|
||||
raise ValueError("Unverified PDF renderer")
|
||||
deadline = time.monotonic() + 20
|
||||
def budget():
|
||||
if time.monotonic() >= deadline:
|
||||
raise ValueError("PDF processing deadline")
|
||||
text, media, seen, indirect = [], [], {}, set()
|
||||
extracted_bytes, characters = 0, 0
|
||||
allowed_streams = set()
|
||||
def append(value):
|
||||
nonlocal characters
|
||||
characters += len(value)
|
||||
text.append(value)
|
||||
if characters > MAX_TEXT:
|
||||
raise ValueError("PDF text budget exceeded")
|
||||
def walk(value, depth=0):
|
||||
nonlocal extracted_bytes
|
||||
budget()
|
||||
if depth > 40 or len(seen) + len(indirect) > MAX_OBJECTS:
|
||||
raise ValueError("PDF object budget exceeded")
|
||||
if isinstance(value, IndirectObject):
|
||||
identity = (value.idnum, value.generation)
|
||||
if identity in indirect: return
|
||||
indirect.add(identity)
|
||||
walk(value.get_object(), depth + 1)
|
||||
return
|
||||
if isinstance(value, (DictionaryObject, ArrayObject)):
|
||||
if id(value) in seen: return
|
||||
# Retain objects as well as IDs: parsed operand containers could
|
||||
# otherwise be collected and their IDs reused across streams.
|
||||
seen[id(value)] = value
|
||||
if isinstance(value, DictionaryObject):
|
||||
if any(str(key) in FORBIDDEN for key in value) or str(value.get("/Type", "")) in ("/Filespec", "/Action", "/ObjStm", "/EmbeddedFile", "/XRef"):
|
||||
raise ValueError("Unsupported active PDF content")
|
||||
if value.get("/Type") == "/Font" and (value.get("/Subtype") != "/Type1" or value.get("/BaseFont") not in STANDARD_FONTS or "/FontDescriptor" in value or "/ToUnicode" in value or value.get("/Encoding") not in (None, "/WinAnsiEncoding", "/MacRomanEncoding", "/StandardEncoding")):
|
||||
raise ValueError("Unsupported PDF font encoding")
|
||||
if isinstance(value, StreamObject):
|
||||
if id(value) not in allowed_streams:
|
||||
raise ValueError("Unsupported hidden PDF stream")
|
||||
decoded = value.get_data()
|
||||
extracted_bytes += len(decoded)
|
||||
if extracted_bytes > MAX_BYTES:
|
||||
raise ValueError("PDF expansion budget exceeded")
|
||||
try: append(decoded.decode("utf-8", errors="strict"))
|
||||
except UnicodeError: pass
|
||||
if value.get("/Subtype") != "/Image":
|
||||
operations = ContentStream(value, reader).operations
|
||||
if len(operations) > MAX_OBJECTS: raise ValueError("PDF operation budget exceeded")
|
||||
for operands, _operator in operations:
|
||||
for operand in operands: walk(operand, depth + 1)
|
||||
for key, item in value.items():
|
||||
append(str(key))
|
||||
walk(item, depth + 1)
|
||||
elif isinstance(value, ArrayObject):
|
||||
for item in value: walk(item, depth + 1)
|
||||
elif isinstance(value, (TextStringObject, NameObject)): append(str(value))
|
||||
elif isinstance(value, ByteStringObject):
|
||||
try: append(bytes(value).decode("utf-8", errors="strict"))
|
||||
except UnicodeError: raise ValueError("Unsupported PDF string encoding")
|
||||
with stage("pdf.structure"):
|
||||
reader = PdfReader(io.BytesIO(data), strict=True)
|
||||
if reader.is_encrypted or reader.trailer.get("/Prev") or not 0 < len(reader.pages) <= MAX_PAGES:
|
||||
raise ValueError("Unsupported PDF document")
|
||||
# Only page content and page-reachable image resources may carry streams.
|
||||
# Fonts/alternate encodings, hidden orphan streams and XMP streams remain
|
||||
# outside this deliberately explicit static PDF subset.
|
||||
def allow_resource(value, depth=0, visited=None):
|
||||
if visited is None: visited = set()
|
||||
budget()
|
||||
if depth > 40 or len(visited) > MAX_OBJECTS: raise ValueError("PDF resource budget exceeded")
|
||||
if isinstance(value, IndirectObject): value = value.get_object()
|
||||
if isinstance(value, (DictionaryObject, ArrayObject)):
|
||||
if id(value) in visited: return
|
||||
visited.add(id(value))
|
||||
if isinstance(value, StreamObject):
|
||||
if value.get("/Subtype") not in ("/Image", "/Form"):
|
||||
raise ValueError("Unsupported PDF resource stream")
|
||||
allowed_streams.add(id(value))
|
||||
if isinstance(value, DictionaryObject):
|
||||
for item in value.values(): allow_resource(item, depth + 1, visited)
|
||||
elif isinstance(value, ArrayObject):
|
||||
for item in value: allow_resource(item, depth + 1, visited)
|
||||
for page in reader.pages:
|
||||
contents = page.get("/Contents")
|
||||
if isinstance(contents, IndirectObject): contents = contents.get_object()
|
||||
if isinstance(contents, ArrayObject):
|
||||
for item in contents:
|
||||
item = item.get_object()
|
||||
if not isinstance(item, StreamObject): raise ValueError("Unsupported PDF page content")
|
||||
allowed_streams.add(id(item))
|
||||
elif isinstance(contents, StreamObject): allowed_streams.add(id(contents))
|
||||
elif contents is not None: raise ValueError("Unsupported PDF page content")
|
||||
if page.get("/Resources") is not None: allow_resource(page["/Resources"])
|
||||
walk(reader.trailer)
|
||||
# Inspect every indexed object, including unreferenced metadata/actions.
|
||||
for generation, entries in reader.xref.items():
|
||||
for number in entries:
|
||||
if number: walk(IndirectObject(number, generation, reader))
|
||||
for number in reader.xref_objStm:
|
||||
walk(IndirectObject(number, 0, reader))
|
||||
media_bytes, pixels = 0, 0
|
||||
with pdfium.PdfDocument(data) as document:
|
||||
if len(document) != len(reader.pages): raise ValueError("PDF parser disagreement")
|
||||
for index, source in enumerate(reader.pages):
|
||||
budget()
|
||||
append(source.extract_text() or "")
|
||||
# Hidden/unpainted image resources also require independent OCR.
|
||||
with stage("pdf.images"):
|
||||
for image in source.images:
|
||||
if len(image.data) > MAX_BYTES: raise ValueError("PDF image budget exceeded")
|
||||
media_bytes += len(image.data)
|
||||
if media_bytes > MAX_BYTES or len(media) >= 64: raise ValueError("PDF media budget exceeded")
|
||||
value, complete = ocr_image(image.data)
|
||||
if not complete: raise ValueError("Incomplete PDF image OCR")
|
||||
append(value)
|
||||
media.append(("image", image.data))
|
||||
page = document[index]
|
||||
try:
|
||||
width, height = page.get_size()
|
||||
if not all(math.isfinite(value) and value > 0 for value in (width, height)):
|
||||
raise ValueError("Invalid PDF page dimensions")
|
||||
count = math.ceil(width * 2) * math.ceil(height * 2)
|
||||
pixels += count
|
||||
if count > MAX_PAGE_PIXELS or pixels > MAX_PIXELS:
|
||||
raise ValueError("PDF rendering budget exceeded")
|
||||
with stage("pdf.render"):
|
||||
bitmap = page.render(scale=2, draw_annots=False)
|
||||
try:
|
||||
rendered = io.BytesIO()
|
||||
bitmap.to_pil().save(rendered, format="PNG")
|
||||
payload = rendered.getvalue()
|
||||
finally: bitmap.close()
|
||||
media_bytes += len(payload)
|
||||
if media_bytes > MAX_BYTES or len(media) >= 64: raise ValueError("PDF rendered media budget exceeded")
|
||||
with stage("pdf.ocr"):
|
||||
value, complete = ocr_image(payload)
|
||||
if not complete: raise ValueError("Incomplete PDF rendered OCR")
|
||||
append(value)
|
||||
media.append(("image", payload))
|
||||
finally: page.close()
|
||||
budget()
|
||||
return "\n".join(text), True, media
|
||||
@@ -0,0 +1,2 @@
|
||||
# Official PyPI Windows x64 wheel, includes PDFium and its license notices.
|
||||
pypdfium2==5.14.0 --hash=sha256:149fd5c6397b8df8bf7911a93506eff0be874f877afe7ac936cf5d37d21a6a06
|
||||
@@ -1,4 +1,5 @@
|
||||
pypdf>=6,<8
|
||||
pypdfium2==5.14.0
|
||||
Pillow>=12,<14
|
||||
defusedxml>=0.7,<1
|
||||
yara-python==4.5.4
|
||||
|
||||
@@ -5,7 +5,7 @@ import math
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
|
||||
STAGES = frozenset(('inspect', 'parse', 'ocr', 'ocr.load', 'yara', 'semantic', 'semantic.load', 'macros'))
|
||||
STAGES = frozenset(('inspect', 'parse', 'ocr', 'ocr.load', 'yara', 'semantic', 'semantic.load', 'macros', 'pdf.structure', 'pdf.images', 'pdf.render', 'pdf.ocr', 'office.archive', 'office.xml', 'office.relationships', 'office.ocr'))
|
||||
_current = contextvars.ContextVar('inspection_timings', default=None)
|
||||
|
||||
def enabled(policy):
|
||||
|
||||
@@ -14,9 +14,12 @@ class InspectionTests(unittest.TestCase):
|
||||
def test_office_extraction(self):
|
||||
output=io.BytesIO()
|
||||
with zipfile.ZipFile(output,"w") as archive:
|
||||
archive.writestr("word/document.xml",'<document xmlns="urn:test"><t>Secret Roadmap</t></document>')
|
||||
archive.writestr("[Content_Types].xml",'<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types"><Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/><Override PartName="/word/document.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/></Types>')
|
||||
archive.writestr("_rels/.rels",'<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships"><Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="word/document.xml"/></Relationships>')
|
||||
archive.writestr("word/document.xml",'<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"><w:body><w:p><w:r><w:t>Secret Roadmap</w:t></w:r></w:p></w:body></w:document>')
|
||||
result=inspect_content(output.getvalue(),policy())
|
||||
self.assertEqual(result["contentType"],"office");self.assertEqual(result["ruleId"],"secret")
|
||||
self.assertTrue(result["coverage"]["complete"])
|
||||
def test_pdf_extraction(self):
|
||||
from pypdf import PdfWriter
|
||||
from pypdf.generic import DecodedStreamObject,NameObject,DictionaryObject
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
import io
|
||||
import unittest
|
||||
import zlib
|
||||
from unittest.mock import patch
|
||||
from pypdf import PdfWriter, PdfReader
|
||||
from pypdf.generic import NameObject, DictionaryObject, DecodedStreamObject, EncodedStreamObject, TextStringObject
|
||||
from engine import inspect_content
|
||||
from pdf_documents import extract_pdf
|
||||
|
||||
|
||||
def policy():
|
||||
return {"dlp": {"maxBytes": 1048576, "rules": [{"id": "secret", "kind": "literal", "value": "Secret Roadmap"}], "semantic": {"enabled": False}}}
|
||||
|
||||
|
||||
def pdf(value=b'BT /F1 24 Tf 20 150 Td (Public information) Tj ET', mutate=None):
|
||||
writer = PdfWriter()
|
||||
page = writer.add_blank_page(400, 220)
|
||||
font = DictionaryObject({NameObject('/Type'): NameObject('/Font'), NameObject('/Subtype'): NameObject('/Type1'), NameObject('/BaseFont'): NameObject('/Helvetica')})
|
||||
page[NameObject('/Resources')] = DictionaryObject({NameObject('/Font'): DictionaryObject({NameObject('/F1'): writer._add_object(font)})})
|
||||
stream = DecodedStreamObject(); stream.set_data(value)
|
||||
page[NameObject('/Contents')] = writer._add_object(stream)
|
||||
if mutate: mutate(writer, page)
|
||||
output = io.BytesIO(); writer.write(output)
|
||||
return output.getvalue()
|
||||
|
||||
|
||||
class PdfTests(unittest.TestCase):
|
||||
def test_public_render_and_ocr_all_pages(self):
|
||||
result = inspect_content(pdf(), policy())
|
||||
self.assertEqual(result['decision'], 'allow')
|
||||
self.assertTrue(result['coverage']['complete'])
|
||||
self.assertEqual(result['coverage']['mediaItems'], 1)
|
||||
|
||||
def test_encoded_secret_decoded_text(self):
|
||||
data = pdf(b'BT /F1 24 Tf 20 150 Td <53656372657420526F61646D6170> Tj ET')
|
||||
self.assertNotIn(b'Secret Roadmap', data)
|
||||
self.assertEqual(inspect_content(data, policy())['ruleId'], 'secret')
|
||||
|
||||
def test_hidden_metadata_and_unreferenced_strings(self):
|
||||
for mutation in [lambda w, _: w.add_metadata({'/Title': 'Secret Roadmap'}), lambda w, _: w._add_object(TextStringObject('Secret Roadmap'))]:
|
||||
self.assertEqual(inspect_content(pdf(mutate=mutation), policy())['ruleId'], 'secret')
|
||||
|
||||
def test_render_only_vector_secret_has_no_extractable_text(self):
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
mask = Image.new('L', (500, 80), 255)
|
||||
ImageDraw.Draw(mask).text((10, 8), 'Secret Roadmap', font=ImageFont.load_default(size=42), fill=0)
|
||||
operations = []
|
||||
for y in range(mask.height):
|
||||
x = 0
|
||||
while x < mask.width:
|
||||
if mask.getpixel((x, y)) < 128:
|
||||
start = x
|
||||
while x < mask.width and mask.getpixel((x, y)) < 128: x += 1
|
||||
operations.append(f'{start + 15} {180-y} {x-start} 1 re f')
|
||||
else: x += 1
|
||||
data = pdf(('\n'.join(operations)).encode())
|
||||
self.assertFalse(PdfReader(io.BytesIO(data)).pages[0].extract_text())
|
||||
self.assertNotIn(b'Secret Roadmap', data)
|
||||
self.assertEqual(inspect_content(data, policy())['ruleId'], 'secret')
|
||||
|
||||
def test_embedded_raster_secret(self):
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
image = Image.new('RGB', (700, 200), 'white')
|
||||
ImageDraw.Draw(image).text((20, 40), 'Secret Roadmap', font=ImageFont.load_default(size=48), fill='black')
|
||||
output = io.BytesIO(); image.save(output, format='PDF')
|
||||
self.assertEqual(inspect_content(output.getvalue(), policy())['ruleId'], 'secret')
|
||||
|
||||
def test_unsupported_active_encrypted_or_hidden_stream(self):
|
||||
def action(writer, _): writer._root_object[NameObject('/OpenAction')] = DictionaryObject({NameObject('/S'): NameObject('/JavaScript')})
|
||||
def attachment(writer, _): writer.add_attachment('private.txt', b'opaque')
|
||||
def encryption(writer, _): writer.encrypt('private')
|
||||
def hidden_stream(writer, _):
|
||||
stream = DecodedStreamObject(); stream.set_data(b'opaque'); writer._add_object(stream)
|
||||
for mutation in [action, attachment, encryption, hidden_stream]:
|
||||
result = inspect_content(pdf(mutate=mutation), policy())
|
||||
self.assertEqual(result['decision'], 'block')
|
||||
self.assertFalse(result['coverage']['complete'])
|
||||
|
||||
def test_decoder_and_render_budgets_fail_before_ocr(self):
|
||||
def bomb(writer, page):
|
||||
stream = EncodedStreamObject(); stream._data = zlib.compress(b'x' * (21 * 1024 * 1024))
|
||||
stream[NameObject('/Filter')] = NameObject('/FlateDecode')
|
||||
reference = page.raw_get('/Contents')
|
||||
stream.indirect_reference = reference
|
||||
writer._objects[reference.idnum - 1] = stream
|
||||
from pypdf.filters import FlateDecode
|
||||
from pypdf import get_configuration
|
||||
original_decode = FlateDecode.decode
|
||||
decodes = []
|
||||
def bounded_decode(*args, **kwargs):
|
||||
decodes.append(get_configuration().zlib_maximum_output_length)
|
||||
return original_decode(*args, **kwargs)
|
||||
with patch('engine.ocr_image', side_effect=AssertionError('Must not OCR oversized input')):
|
||||
with patch.object(FlateDecode, 'decode', side_effect=bounded_decode):
|
||||
self.assertEqual(inspect_content(pdf(mutate=bomb), policy())['decision'], 'block')
|
||||
self.assertEqual(decodes, [20 * 1024 * 1024])
|
||||
self.assertEqual(inspect_content(pdf(mutate=lambda _w, page: setattr(page.mediabox, 'right', 1000000)), policy())['decision'], 'block')
|
||||
|
||||
def test_missing_ocr_never_claims_complete(self):
|
||||
with self.assertRaises(ValueError): extract_pdf(pdf(), lambda _data: ('', False))
|
||||
|
||||
def test_debug_stages_record_actual_structure_render_and_ocr(self):
|
||||
from stage_timings import collect, metadata, validated
|
||||
p = policy(); p['diagnostics'] = {'enabled': True, 'mode': 'debug'}
|
||||
with collect(p) as state:
|
||||
result = inspect_content(pdf(), p)
|
||||
self.assertEqual(result['decision'], 'allow')
|
||||
envelope = validated(metadata(state))
|
||||
self.assertIsNotNone(envelope)
|
||||
stages = {record['stage'] for record in envelope['records']}
|
||||
self.assertTrue({'pdf.structure', 'pdf.images', 'pdf.render', 'pdf.ocr'} <= stages)
|
||||
for record in envelope['records']:
|
||||
self.assertGreaterEqual(record['durationMs'], 0)
|
||||
if record['stage'].startswith('pdf.'): self.assertIsNotNone(record['parentId'])
|
||||
|
||||
|
||||
if __name__ == '__main__': unittest.main()
|
||||
Reference in new issue
Block a user