Inspect static PDF and OOXML content with bounded local coverage
This commit is contained in:
Karl committed
1 parent
69d820e315
commit
9e7cb57093
14 files changed
+758
-51
No files matched your search
@@ -4,7 +4,8 @@ All inspection is local. No scan downloads models or calls cloud inference.
|
||||
|
||||
- YARA: genuine pinned yara-python 4.5.4 in private Windows CPython 3.13 runtime. Centrally supplied sources have bounded size and count, includes disabled, imported modules disallowed, compile warnings treated as errors, and five-second match timeout. Rules compile from in-memory strings and scan bytes. Errors, unavailable engine, missing rules and oversize payloads block downloads. Compilation currently runs inside the inspector process; a native compiler hang is bounded by the browser's inspector request deadline but does not restart the inspector automatically.
|
||||
- Text: UTF-8 extraction and deterministic secret/literal/fingerprint classification.
|
||||
- PDF/OOXML: bounded extraction of text and embedded images; full rendered visual coverage is incomplete and outbound DLP fails closed.
|
||||
- PDF: bounded static subset now inspects decoded text, metadata and all indexed objects, independently OCRs embedded page images, and renders/OCRs every page with the bundled offline PDFium/OCR runtime. Active content, forms, annotations, attachments, layers, incremental updates, orphan/unsupported streams and embedded/custom font encodings remain blocked. See [PDF coverage](PDF-COVERAGE.md).
|
||||
- OOXML: bounded extraction of text and embedded images; full rendered visual coverage is incomplete and outbound DLP fails closed.
|
||||
- Images: bundled RapidOCR 3.8.1 with CPU ONNXRuntime 1.24.4 and packaged PP-OCRv4 detection/recognition plus v2 classification weights. Explicit verified local model paths bypass all model downloads. Real Windows test renders a secret into an image and blocks it with network model downloading forbidden. Models primarily support Chinese/English; other languages and recognition accuracy require validation. OCR completion indicates all frames processed, not guaranteed recognition accuracy. Pillow parsing has dimension/frame limits. Optional `SAFETYNET_OCR_ENGINE=tesseract` uses administrator-installed trusted `TESSERACT_PATH`, optionally `TESSDATA_PREFIX` for local traineddata. Tesseract weights are not bundled. Missing OCR blocks DLP.
|
||||
- Audio/video: recognition of container types only. Transcription and full video inspection are unconfigured; DLP blocks rather than claiming safe coverage.
|
||||
- Semantic adapter: real Google EmbeddingGemma 2, Q8_0 GGUF export revision `bfcd298762cc34d0357ece5ebdd31791a3a374d8`, 768 dimensions. A verified native worker runs explicitly on CPU, reads bounded local NDJSON, and returns actual normalized vectors. Both GGUF weights and native runtime modules must pass size/SHA-256 checks before loading. Configure `EMBEDDING_MODEL_DIR` and `EMBEDDING_WORKER_PATH` to staged, protected local resources. No cloud inference, model download or Python ML framework is used. Text uses the same pinned `document-v1` profile for training and inspection; native token windows must cover exact UTF-8 bytes through the final byte, otherwise scanning fails closed. At most 64 vectors are allowed; incomplete or oversized training is refused. Native image embeddings process all bounded frames resized to at most672px with48px alignment. Native audio supports only mono16kHz PCM16 WAV up to30seconds; other media remains uninspectable. Actual model fixtures must run separately from protocol unit tests to establish operational readiness.
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
# Local static PDF coverage
|
||||
|
||||
The inspector uses hash-pinned `pypdfium2 5.14.0` with official Windows x64 PDFium `156.0.8076.0`. The wheel has no V8 or XFA feature flags. No JavaScript, forms, external URLs or network rendering are enabled. This does not modify Chromium or Electron.
|
||||
|
||||
For the supported subset, inspection combines decoded page text, every indexed object's strings and metadata, decoded content operands, independent OCR of original page image resources, and rendering/OCR of every page. Rendered pages and original images are also supplied to local semantic analysis when policy requires it. Rendering alone is never treated as coverage of hidden bytes.
|
||||
|
||||
Limits include 32 pages, 10,000 objects/operations, depth 40, 20 million pixels per rendered page, 50 million rendered pixels total, 20 MiB decoded streams/media, 64 media items, one million extracted characters and a 20-second cooperative PDF deadline. The existing isolated worker enforces an external deadline and remains in the inspector's 4 GiB Windows Job Object. Decoder output, declared lengths, filter chains and decoding work are bounded before decompression via the installed pypdf configuration API. The external process deadline remains necessary for native rendering/OCR calls.
|
||||
|
||||
The supported font subset is the 14 standard PDF Type 1 fonts with standard, WinAnsi or MacRoman encoding. Embedded fonts, custom encoding/CMaps, active actions, annotations, forms, optional layers, attachments, incremental revisions, hidden/orphan streams and unsupported resources fail closed. This is **not full compatibility with common PDF exports**, which often embed fonts or use XMP metadata. Broader support needs independent coverage tests rather than changing the completeness flag.
|
||||
|
||||
`coverage.complete` means every component in this explicit processing subset was inspected; OCR and classification remain probabilistic and can miss visually ambiguous information. It does not certify that a document is harmless.
|
||||
|
||||
Actual offline tests cover benign PDF release, hex-encoded text secrets, metadata and unreferenced strings, vector-only rendered lettering without extractable text, embedded raster secrets, active/encrypted/hidden content rejection and a compressed expansion bomb rejected before OCR. These tests use the real renderer and local OCR, apart from the explicit missing-OCR failure test.
|
||||
|
||||
PDF timing stages are fixed `pdf.structure`, `pdf.images`, `pdf.render` and `pdf.ocr`, under the existing private `parse`/`inspect` stages. They contain timings and correlation, never document text.
|
||||
|
||||
Sources: [pypdfium2 release and licensing](https://pypi.org/project/pypdfium2/5.14.0/), [official PDFium API and process-safety guidance](https://pypdfium2.readthedocs.io/en/stable/python_api.html). The runtime retains the wheel's BSD/Apache and PDFium dependency license notices in its distribution metadata.
|
||||
@@ -200,51 +200,12 @@ def extract(data, kind):
|
||||
text,complete=ocr_image(data)
|
||||
return text,complete,[("image",data)]
|
||||
if kind=="pdf":
|
||||
from pypdf import PdfReader
|
||||
reader=PdfReader(io.BytesIO(data),strict=True)
|
||||
if reader.is_encrypted: raise ValueError("Encrypted PDF cannot be inspected")
|
||||
if len(reader.pages)>100: raise ValueError("PDF page budget exceeded")
|
||||
text=[];complete=True;media=[]
|
||||
for page in reader.pages:
|
||||
value=page.extract_text() or ""
|
||||
text.append(value)
|
||||
images=list(page.images)
|
||||
if not value.strip() and not images: complete=False
|
||||
for image in images:
|
||||
if len(image.data)>MAX_BYTES: raise ValueError("PDF image budget exceeded")
|
||||
scanned,ok=ocr_image(image.data);text.append(scanned);complete=complete and ok
|
||||
media.append(("image",image.data))
|
||||
if len(media)>100: raise ValueError("PDF media budget exceeded")
|
||||
# Attachments, JavaScript, forms, actions and vector drawings need deeper inspection.
|
||||
if page.get("/Annots"): complete=False
|
||||
root=reader.trailer["/Root"]
|
||||
if root.get("/AcroForm") or root.get("/OpenAction") or root.get("/Names"): complete=False
|
||||
# Text extraction is useful, but does not prove all visible/vector PDF content was covered.
|
||||
return "\n".join(text),False,media
|
||||
from pdf_documents import extract_pdf
|
||||
return extract_pdf(data, ocr_image)
|
||||
if kind=="office":
|
||||
from defusedxml.ElementTree import fromstring
|
||||
text=[];media=[];complete=True
|
||||
with zipfile.ZipFile(io.BytesIO(data)) as archive:
|
||||
members=archive.infolist()
|
||||
if len(members)>2000 or sum(x.file_size for x in members)>MAX_BYTES:
|
||||
raise ValueError("Archive expansion budget exceeded")
|
||||
if not any(x.filename in ["word/document.xml","ppt/presentation.xml","xl/workbook.xml"] for x in members):
|
||||
raise ValueError("Only OOXML document archives are supported")
|
||||
for entry in members:
|
||||
if entry.flag_bits&1 or entry.file_size>MAX_BYTES or entry.file_size>max(entry.compress_size,1)*200:
|
||||
raise ValueError("Unsafe document archive")
|
||||
name=entry.filename
|
||||
if any(s in name.lower() for s in ["vbaproject","embeddings/","activex/","customxml/"]): complete=False
|
||||
if name.endswith(".xml"):
|
||||
root=fromstring(archive.read(entry))
|
||||
text.extend(element.text for element in root.iter() if element.tag.rsplit("}",1)[-1] in ["t","v"] and element.text)
|
||||
elif "/media/" in name:
|
||||
payload=archive.read(entry)
|
||||
if content_kind(payload)!="image": complete=False; continue
|
||||
scanned,ok=ocr_image(payload);text.append(scanned);complete=complete and ok;media.append(("image",payload))
|
||||
if len(media)>100: raise ValueError("Office media budget exceeded")
|
||||
# Charts, relationships, equations and rendered layout are not yet fully inspected.
|
||||
return "\n".join(text),False,media
|
||||
from office_content import extract_office
|
||||
result=extract_office(data,ocr_image,max_bytes=MAX_BYTES)
|
||||
return result["text"],result["complete"],result["media"]
|
||||
if kind in ["audio","video"]: return "",False,[(kind,data)]
|
||||
raise ValueError("Unsupported content type")
|
||||
|
||||
|
||||
@@ -0,0 +1,216 @@
|
||||
"""Bounded static OOXML content coverage, not an Office renderer or macro engine.
|
||||
|
||||
OPC and part families: Microsoft Open XML SDK structure documentation:
|
||||
https://learn.microsoft.com/en-us/office/open-xml/general/how-to-create-a-package
|
||||
https://learn.microsoft.com/en-us/office/open-xml/spreadsheet/structure-of-a-spreadsheetml-document
|
||||
https://learn.microsoft.com/en-us/office/open-xml/presentation/structure-of-a-presentationml-document
|
||||
All XML text, tails, attributes, namespace declarations and ZIP metadata are inputs.
|
||||
Only known static XML parts and successfully OCR-inspected raster assets are covered.
|
||||
Unknown parts, equations, vector/custom layouts and active features remain incomplete.
|
||||
"""
|
||||
import io
|
||||
import posixpath
|
||||
import re
|
||||
import struct
|
||||
import unicodedata
|
||||
import zipfile
|
||||
from urllib.parse import urlsplit
|
||||
from defusedxml.ElementTree import fromstring,iterparse
|
||||
from stage_timings import stage
|
||||
CT='http://schemas.openxmlformats.org/package/2006/content-types'
|
||||
REL='http://schemas.openxmlformats.org/package/2006/relationships'
|
||||
R='http://schemas.openxmlformats.org/officeDocument/2006/relationships'
|
||||
W='http://schemas.openxmlformats.org/wordprocessingml/2006/main'
|
||||
S='http://schemas.openxmlformats.org/spreadsheetml/2006/main'
|
||||
P='http://schemas.openxmlformats.org/presentationml/2006/main'
|
||||
A='http://schemas.openxmlformats.org/drawingml/2006/main'
|
||||
CP='http://schemas.openxmlformats.org/package/2006/metadata/core-properties'
|
||||
EP='http://schemas.openxmlformats.org/officeDocument/2006/extended-properties'
|
||||
CUSTOM='http://schemas.openxmlformats.org/officeDocument/2006/custom-properties'
|
||||
TYPE_PREFIX='application/vnd.openxmlformats-officedocument.'
|
||||
MAIN={TYPE_PREFIX+'wordprocessingml.document.main+xml':('docx',W,'document'),TYPE_PREFIX+'spreadsheetml.sheet.main+xml':('xlsx',S,'workbook'),TYPE_PREFIX+'presentationml.presentation.main+xml':('pptx',P,'presentation')}
|
||||
XML_TYPES={TYPE_PREFIX+'wordprocessingml.'+name+'+xml':(W,root) for name,root in [('styles','styles'),('settings','settings'),('fontTable','fonts'),('numbering','numbering'),('header','hdr'),('footer','ftr'),('comments','comments'),('footnotes','footnotes'),('endnotes','endnotes'),('webSettings','webSettings')]}
|
||||
XML_TYPES.update({TYPE_PREFIX+'spreadsheetml.'+name+'+xml':(S,root) for name,root in [('worksheet','worksheet'),('sharedStrings','sst'),('styles','styleSheet'),('table','table')]})
|
||||
XML_TYPES.update({TYPE_PREFIX+'presentationml.'+name+'+xml':(P,root) for name,root in [('slide','sld'),('notesSlide','notes'),('notesMaster','notesMaster'),('slideMaster','sldMaster'),('slideLayout','sldLayout'),('presProps','presentationPr'),('viewProps','viewPr'),('tableStyles','tblStyleLst')]})
|
||||
XML_TYPES['application/vnd.ms-word.stylesWithEffects+xml']=(W,'styles')
|
||||
XML_TYPES[TYPE_PREFIX+'customXmlProperties+xml']=('http://schemas.openxmlformats.org/officeDocument/2006/customXml','datastoreItem')
|
||||
XML_TYPES[TYPE_PREFIX+'presentationml.tableStyles+xml']=(A,'tblStyleLst')
|
||||
XML_TYPES.update({'application/vnd.openxmlformats-package.core-properties+xml':(CP,'coreProperties'),TYPE_PREFIX+'extended-properties+xml':(EP,'Properties'),TYPE_PREFIX+'custom-properties+xml':(CUSTOM,'Properties'),TYPE_PREFIX+'theme+xml':(A,'theme')})
|
||||
SAFE_REL={R+'/'+name for name in ['officeDocument','extended-properties','custom-properties','styles','settings','fontTable','numbering','header','footer','comments','footnotes','endnotes','worksheet','sharedStrings','table','theme','slide','notesSlide','notesMaster','slideMaster','slideLayout','presProps','viewProps','tableStyles','image']}
|
||||
SAFE_REL.update({R+'/customXml',R+'/customXmlProps',R+'/webSettings','http://schemas.microsoft.com/office/2007/relationships/stylesWithEffects','http://schemas.openxmlformats.org/package/2006/relationships/metadata/core-properties','http://schemas.openxmlformats.org/package/2006/relationships/metadata/thumbnail'})
|
||||
SAFE_NS={CT,REL,R,W,S,P,A,CP,EP,CUSTOM,'http://schemas.openxmlformats.org/drawingml/2006/picture','http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing','http://schemas.openxmlformats.org/officeDocument/2006/docPropsVTypes','http://purl.org/dc/elements/1.1/','http://purl.org/dc/terms/','http://purl.org/dc/dcmitype/','http://www.w3.org/2001/XMLSchema-instance','http://www.w3.org/XML/1998/namespace','http://schemas.openxmlformats.org/markup-compatibility/2006','http://schemas.openxmlformats.org/officeDocument/2006/bibliography','http://schemas.openxmlformats.org/officeDocument/2006/customXml','http://schemas.openxmlformats.org/officeDocument/2006/math'}
|
||||
SAFE_EXTENSION_TAGS={'{urn:schemas-microsoft-com:office:office}'+name for name in ['shapedefaults','shapelayout','idmap']}|{'{http://schemas.microsoft.com/office/word/2010/wordml}'+name for name in ['docId','defaultImageDpi']}
|
||||
UNSUPPORTED_TAGS={'AlternateContent','oMath','oMathPara','custGeom','pict','object','control','oleObj','graphicFrame','chart','grpSp','cxnSp','fldSimple','fldChar','instrText','altChunk','externalLink','externalReference','externalReferences','blob','hlinkClick','hlinkHover','animation','transition','timing','embedRegular','embedBold','embedItalic','embedBoldItalic'}
|
||||
class OfficeCoverageError(ValueError):pass
|
||||
|
||||
def _part(name):
|
||||
if not isinstance(name,str) or not name or len(name)>240 or '\\' in name or '%' in name or name.startswith('/') or any(part in {'','.', '..'} for part in name.split('/')) or any(ord(c)<32 for c in name) or unicodedata.normalize('NFC',name)!=name:raise OfficeCoverageError('office-part-name-invalid')
|
||||
return name
|
||||
|
||||
def _physical_zip(data,archive,infos):
|
||||
if not data.startswith(b'PK\x03\x04'):raise OfficeCoverageError('office-archive-prefix-invalid')
|
||||
cursor=0
|
||||
for info in sorted(infos,key=lambda item:item.header_offset):
|
||||
if info.header_offset!=cursor or cursor+30>len(data):raise OfficeCoverageError('office-archive-gap')
|
||||
signature,version,flags,method,time,date,crc,compressed,size,name_length,extra_length=struct.unpack_from('<IHHHHHIIIHH',data,cursor)
|
||||
if signature!=0x04034b50 or flags!=info.flag_bits or method!=info.compress_type:raise OfficeCoverageError('office-local-header-invalid')
|
||||
if extra_length:raise OfficeCoverageError('office-local-extra-uncovered')
|
||||
cursor+=30+name_length+extra_length+info.compress_size
|
||||
if flags&8:
|
||||
if data[cursor:cursor+4]==b'PK\x07\x08':cursor+=4
|
||||
if cursor+12>len(data) or struct.unpack_from('<III',data,cursor)!=(info.CRC,info.compress_size,info.file_size):raise OfficeCoverageError('office-descriptor-invalid')
|
||||
cursor+=12
|
||||
elif (crc,compressed,size)!=(info.CRC,info.compress_size,info.file_size):raise OfficeCoverageError('office-local-size-invalid')
|
||||
if cursor!=archive.start_dir:raise OfficeCoverageError('office-archive-gap')
|
||||
for _info in infos:
|
||||
if data[cursor:cursor+4]!=b'PK\x01\x02' or cursor+46>len(data):raise OfficeCoverageError('office-central-header-invalid')
|
||||
name_length,extra_length,comment_length=struct.unpack_from('<HHH',data,cursor+28);cursor+=46+name_length+extra_length+comment_length
|
||||
if data[cursor:cursor+4]!=b'PK\x05\x06' or cursor+22>len(data):raise OfficeCoverageError('office-end-record-invalid')
|
||||
disk,start_disk,count_disk,count,total_size,start,comment=struct.unpack_from('<HHHHIIH',data,cursor+4)
|
||||
if disk or start_disk or count_disk!=len(infos) or count!=len(infos) or start!=archive.start_dir or total_size!=cursor-archive.start_dir or cursor+22+comment!=len(data):raise OfficeCoverageError('office-end-record-invalid')
|
||||
|
||||
def extract_office(data,ocr_image,*,max_bytes=20*1024*1024,max_parts=2000,max_media=100,max_text=2*1024*1024):
|
||||
if not isinstance(data,bytes) or len(data)>max_bytes or not callable(ocr_image):raise OfficeCoverageError('office-input-invalid')
|
||||
texts=[];seen_text=set();media=[];reasons=set();xml={};images={};types={};members={};payloads={};total=0;nodes=0
|
||||
def append(value):
|
||||
nonlocal total
|
||||
if value:
|
||||
value=str(value)
|
||||
if value in seen_text:return
|
||||
seen_text.add(value);total+=len(value)
|
||||
if total>max_text:raise OfficeCoverageError('office-text-limit')
|
||||
texts.append(value)
|
||||
with stage('office.archive'):
|
||||
with zipfile.ZipFile(io.BytesIO(data)) as archive:
|
||||
infos=archive.infolist()
|
||||
_physical_zip(data,archive,infos)
|
||||
if len(infos)>max_parts or sum(item.file_size for item in infos)>max_bytes:raise OfficeCoverageError('office-expansion-limit')
|
||||
append(archive.comment.decode('utf-8','strict'))
|
||||
for info in infos:
|
||||
name=_part(info.filename)
|
||||
if name.casefold() in members:raise OfficeCoverageError('office-duplicate-part')
|
||||
members[name.casefold()]=(name,info)
|
||||
if info.is_dir() or info.flag_bits&1 or info.compress_type not in {zipfile.ZIP_STORED,zipfile.ZIP_DEFLATED} or info.file_size>max_bytes or info.file_size>max(1,info.compress_size)*200 or (info.external_attr>>16)&0o170000==0o120000:raise OfficeCoverageError('office-archive-unsafe')
|
||||
# Unknown local/central extra fields may hide material outside parts.
|
||||
if info.extra:reasons.add('zip-extra-field-uncovered')
|
||||
append(name);append(info.comment.decode('utf-8','strict'))
|
||||
payload=archive.read(info)
|
||||
if len(payload)!=info.file_size:raise OfficeCoverageError('office-part-size-invalid')
|
||||
payloads[name]=payload
|
||||
with stage('office.xml'):
|
||||
for name,payload in payloads.items():
|
||||
if name.endswith(('.xml','.rels')) or name=='[Content_Types].xml':
|
||||
root=fromstring(payload,forbid_dtd=True,forbid_entities=True,forbid_external=True);xml[name]=root
|
||||
for _event,namespace in iterparse(io.BytesIO(payload),events=['start-ns'],forbid_dtd=True,forbid_entities=True,forbid_external=True):append(namespace[0]);append(namespace[1])
|
||||
for event,item in iterparse(io.BytesIO(payload),events=['comment','pi'],forbid_dtd=True,forbid_entities=True,forbid_external=True):
|
||||
append(item.text)
|
||||
if event=='pi':reasons.add('xml-processing-instruction-unsupported')
|
||||
parents={child:parent for parent in root.iter() for child in parent}
|
||||
append(''.join(value for value in root.itertext() if value.strip()))
|
||||
for element in root.iter():
|
||||
nodes+=1
|
||||
if nodes>100000:raise OfficeCoverageError('office-xml-node-limit')
|
||||
append(element.tag);append(element.text);append(element.tail)
|
||||
for key,value in element.attrib.items():
|
||||
append(key);append(value)
|
||||
if key.startswith('{') and key[1:].split('}',1)[0] not in SAFE_NS and key not in {'{http://schemas.microsoft.com/office/word/2010/wordml}paraId','{http://schemas.microsoft.com/office/word/2010/wordml}textId'} and not (element.tag in SAFE_EXTENSION_TAGS and ((key=='{urn:schemas-microsoft-com:vml}ext' and value=='edit') or (key=='{http://schemas.microsoft.com/office/word/2010/wordml}val' and value.isdecimal()))):reasons.add('xml-namespace-unsupported')
|
||||
namespace=element.tag[1:].split('}',1)[0] if element.tag.startswith('{') else ''
|
||||
if namespace not in SAFE_NS and element.tag not in SAFE_EXTENSION_TAGS:reasons.add('xml-namespace-unsupported')
|
||||
if element.tag.rsplit('}',1)[-1] in UNSUPPORTED_TAGS:reasons.add('xml-feature-unsupported')
|
||||
if element.tag=='{'+A+'}prstGeom' and element.get('prst') not in {'rect','roundRect'}:reasons.add('vector-preset-unsupported')
|
||||
if element.tag=='{'+S+'}f':
|
||||
formula=element.text or '';cached=parents.get(element)
|
||||
cached=cached.find('{'+S+'}v') if cached is not None else None
|
||||
without_strings=re.sub(r'"(?:[^"]|"")*"','',formula)
|
||||
functions={name.upper() for name in re.findall(r'([A-Za-z_][A-Za-z0-9_.]*)\s*\(',without_strings)}
|
||||
safe_functions={'SUM','SUMIF','SUMIFS','COUNT','COUNTIF','COUNTIFS','COUNTA','AVERAGE','AVERAGEIF','AVERAGEIFS','MIN','MAX','ROUND','ROUNDDOWN','ROUNDUP','ABS','IF','AND','OR','NOT'}
|
||||
if not formula or set(element.attrib)-{'ca'} or any(c in without_strings for c in '[]|') or not functions<=safe_functions or cached is None or not (cached.text or '').strip():reasons.add('formula-coverage-incomplete')
|
||||
|
||||
elif payload.startswith((b'\x89PNG\r\n\x1a\n',b'\xff\xd8\xff',b'GIF87a',b'GIF89a')):images[name]=payload
|
||||
else:reasons.add('binary-part-unsupported')
|
||||
with stage('office.relationships'):
|
||||
content=xml.get('[Content_Types].xml')
|
||||
if content is None or content.tag!='{'+CT+'}Types':raise OfficeCoverageError('office-content-types-invalid')
|
||||
defaults={};overrides={}
|
||||
for item in content:
|
||||
if item.tag=='{'+CT+'}Default' and set(item.attrib)=={'Extension','ContentType'}:
|
||||
ext=item.attrib['Extension'].lower()
|
||||
if not re.fullmatch('[a-z0-9]+',ext) or ext in defaults:raise OfficeCoverageError('office-content-types-invalid')
|
||||
defaults[ext]=item.attrib['ContentType']
|
||||
elif item.tag=='{'+CT+'}Override' and set(item.attrib)=={'PartName','ContentType'}:
|
||||
part=item.attrib['PartName']
|
||||
if not part.startswith('/'):raise OfficeCoverageError('office-content-types-invalid')
|
||||
part=_part(part[1:])
|
||||
if part.casefold() in {p.casefold() for p in overrides}:raise OfficeCoverageError('office-content-types-invalid')
|
||||
overrides[part]=item.attrib['ContentType']
|
||||
else:raise OfficeCoverageError('office-content-types-invalid')
|
||||
if any(name.casefold() not in members for name in overrides):raise OfficeCoverageError('office-content-types-dangling')
|
||||
main=[]
|
||||
for name,_info in members.values():
|
||||
if name=='[Content_Types].xml':continue
|
||||
type_=overrides.get(name,defaults.get(name.rsplit('.',1)[-1].lower()))
|
||||
if not type_:raise OfficeCoverageError('office-content-type-missing')
|
||||
types[name]=type_
|
||||
if type_ in MAIN:
|
||||
family,namespace,rootname=MAIN[type_];main.append((name,family))
|
||||
if name not in xml or xml[name].tag!='{'+namespace+'}'+rootname:raise OfficeCoverageError('office-main-part-invalid')
|
||||
elif name.endswith('.rels'):
|
||||
if type_!='application/vnd.openxmlformats-package.relationships+xml':raise OfficeCoverageError('office-relationship-type-invalid')
|
||||
elif type_=='application/xml' and name in xml and xml[name].tag=='{http://schemas.openxmlformats.org/officeDocument/2006/bibliography}Sources':pass
|
||||
elif type_ in XML_TYPES:
|
||||
namespace,rootname=XML_TYPES[type_]
|
||||
if name not in xml or xml[name].tag!='{'+namespace+'}'+rootname:raise OfficeCoverageError('office-xml-part-invalid')
|
||||
elif name in images:
|
||||
expected='image/png' if images[name].startswith(b'\x89PNG') else 'image/jpeg' if images[name].startswith(b'\xff\xd8') else 'image/gif'
|
||||
if type_!=expected:raise OfficeCoverageError('office-image-type-invalid')
|
||||
else:reasons.add('content-type-unsupported')
|
||||
if len(main)!=1:raise OfficeCoverageError('office-main-count-invalid')
|
||||
relationships={};top_main=[]
|
||||
for name,root in xml.items():
|
||||
if not name.endswith('.rels'):continue
|
||||
if root.tag!='{'+REL+'}Relationships':raise OfficeCoverageError('office-relationships-invalid')
|
||||
if name=='_rels/.rels':owner=''
|
||||
elif '/_rels/' in name:
|
||||
prefix,file=name.rsplit('/_rels/',1);owner=prefix+'/'+file[:-5]
|
||||
if owner.casefold() not in members:raise OfficeCoverageError('office-relationships-owner-invalid')
|
||||
else:raise OfficeCoverageError('office-relationships-location-invalid')
|
||||
ids=set()
|
||||
for item in root:
|
||||
if item.tag!='{'+REL+'}Relationship' or not {'Id','Type','Target'}<=set(item.attrib) or set(item.attrib)-{'Id','Type','Target','TargetMode'}:raise OfficeCoverageError('office-relationship-invalid')
|
||||
id_=item.attrib['Id'];target=item.attrib['Target'];type_=item.attrib['Type']
|
||||
if not id_ or id_ in ids:raise OfficeCoverageError('office-relationship-id-invalid')
|
||||
ids.add(id_)
|
||||
if item.attrib.get('TargetMode','Internal')!='Internal':reasons.add('external-relationship-unsupported');continue
|
||||
if urlsplit(target).scheme or '%' in target or '\\' in target or '#' in target or '?' in target:raise OfficeCoverageError('office-relationship-target-invalid')
|
||||
resolved=posixpath.normpath(posixpath.join(posixpath.dirname(owner),target)) if not target.startswith('/') else target[1:]
|
||||
_part(resolved)
|
||||
if resolved.casefold() not in members:raise OfficeCoverageError('office-relationship-target-missing')
|
||||
if type_ not in SAFE_REL:reasons.add('relationship-type-unsupported')
|
||||
else:
|
||||
target_type=types.get(resolved)
|
||||
role=type_.rsplit('/',1)[-1]
|
||||
if role=='officeDocument':valid=target_type in MAIN
|
||||
elif role in {'image','thumbnail'}:valid=resolved in images
|
||||
elif role=='customXml':valid=target_type=='application/xml' and resolved in xml and xml[resolved].tag=='{http://schemas.openxmlformats.org/officeDocument/2006/bibliography}Sources'
|
||||
elif role=='customXmlProps':valid=target_type==TYPE_PREFIX+'customXmlProperties+xml'
|
||||
elif role=='stylesWithEffects':valid=target_type=='application/vnd.ms-word.stylesWithEffects+xml'
|
||||
elif role=='core-properties':valid=target_type=='application/vnd.openxmlformats-package.core-properties+xml'
|
||||
elif role in {'extended-properties','custom-properties'}:valid=target_type==TYPE_PREFIX+role+'+xml'
|
||||
else:valid=target_type in XML_TYPES and (target_type==TYPE_PREFIX+'theme+xml' if role=='theme' else target_type.endswith('.'+role+'+xml'))
|
||||
if not valid:raise OfficeCoverageError('office-relationship-type-target-mismatch')
|
||||
if type_==R+'/officeDocument' and not owner:top_main.append(resolved)
|
||||
relationships[owner]=ids
|
||||
if top_main!=[main[0][0]]:raise OfficeCoverageError('office-root-relationship-invalid')
|
||||
for name,root in xml.items():
|
||||
if name.endswith('.rels') or name=='[Content_Types].xml':continue
|
||||
for element in root.iter():
|
||||
for key,value in element.attrib.items():
|
||||
if key.startswith('{'+R+'}') and value not in relationships.get(name,set()):raise OfficeCoverageError('office-dangling-relationship-reference')
|
||||
with stage('office.ocr'):
|
||||
for name,payload in images.items():
|
||||
if len(media)>=max_media:raise OfficeCoverageError('office-media-limit')
|
||||
scanned,complete=ocr_image(payload)
|
||||
if not isinstance(scanned,str) or complete is not True:reasons.add('image-ocr-incomplete')
|
||||
else:append(scanned)
|
||||
media.append(('image',payload))
|
||||
return {'text':'\n'.join(texts),'complete':not reasons,'media':media,'family':main[0][1],'reasons':sorted(reasons),'parts':len(members)}
|
||||
@@ -0,0 +1,186 @@
|
||||
"""Bounded static PDF inspection: decoded content plus every-page rendering/OCR.
|
||||
|
||||
Active, layered, annotated, embedded-file and incremental documents are outside
|
||||
the supported subset and fail closed. This runs inside the existing isolated
|
||||
parser process; PDFium has no JavaScript/XFA support and forms are never enabled.
|
||||
"""
|
||||
import io
|
||||
import math
|
||||
import time
|
||||
from stage_timings import stage
|
||||
|
||||
MAX_PAGES = 32
|
||||
MAX_OBJECTS = 10000
|
||||
MAX_PIXELS = 50_000_000
|
||||
MAX_PAGE_PIXELS = 20_000_000
|
||||
MAX_BYTES = 20 * 1024 * 1024
|
||||
MAX_TEXT = 1_000_000
|
||||
FORBIDDEN = frozenset(("/Annots", "/AcroForm", "/XFA", "/OpenAction", "/AA", "/A", "/JS", "/JavaScript", "/Names", "/EmbeddedFiles", "/EF", "/AF", "/OCProperties", "/Collection", "/RichMedia", "/Movie", "/Sound", "/3D", "/Rendition", "/Launch", "/URI", "/GoToR", "/GoToE", "/SubmitForm", "/ImportData", "/Encrypt", "/PieceInfo"))
|
||||
STANDARD_FONTS = frozenset(("/Helvetica", "/Helvetica-Bold", "/Helvetica-Oblique", "/Helvetica-BoldOblique", "/Times-Roman", "/Times-Bold", "/Times-Italic", "/Times-BoldItalic", "/Courier", "/Courier-Bold", "/Courier-Oblique", "/Courier-BoldOblique", "/Symbol", "/ZapfDingbats"))
|
||||
|
||||
|
||||
def extract_pdf(data, ocr_image):
|
||||
from pypdf import apply_configuration
|
||||
# Enforce decompressor limits BEFORE any stream get_data()/image allocation.
|
||||
with apply_configuration(maximum_declared_stream_length=MAX_BYTES,
|
||||
array_based_stream_maximum_output_length=MAX_BYTES,
|
||||
zlib_maximum_output_length=MAX_BYTES,
|
||||
lzw_maximum_output_length=MAX_BYTES,
|
||||
run_length_maximum_output_length=MAX_BYTES,
|
||||
brotli_maximum_output_length=MAX_BYTES,
|
||||
jbig2_maximum_output_length=MAX_BYTES,
|
||||
image_maximum_buffer_size=MAX_BYTES,
|
||||
stream_filters_maximum_length=4,
|
||||
stream_decoding_work_maximum_length=MAX_BYTES,
|
||||
zlib_maximum_recovery_input_length=1048576):
|
||||
return _extract_pdf(data, ocr_image)
|
||||
|
||||
|
||||
def _extract_pdf(data, ocr_image):
|
||||
from pypdf import PdfReader
|
||||
from pypdf.generic import IndirectObject, DictionaryObject, ArrayObject, StreamObject, TextStringObject, ByteStringObject, NameObject, ContentStream
|
||||
import pypdfium2 as pdfium
|
||||
if len(data) > MAX_BYTES or data.count(b"%%EOF") != 1 or data.rsplit(b"%%EOF", 1)[-1].strip():
|
||||
raise ValueError("Unsupported PDF container")
|
||||
if pdfium.PDFIUM_INFO.flags or pdfium.PYPDFIUM_INFO.api_tag != (5, 14, 0):
|
||||
raise ValueError("Unverified PDF renderer")
|
||||
deadline = time.monotonic() + 20
|
||||
def budget():
|
||||
if time.monotonic() >= deadline:
|
||||
raise ValueError("PDF processing deadline")
|
||||
text, media, seen, indirect = [], [], {}, set()
|
||||
extracted_bytes, characters = 0, 0
|
||||
allowed_streams = set()
|
||||
def append(value):
|
||||
nonlocal characters
|
||||
characters += len(value)
|
||||
text.append(value)
|
||||
if characters > MAX_TEXT:
|
||||
raise ValueError("PDF text budget exceeded")
|
||||
def walk(value, depth=0):
|
||||
nonlocal extracted_bytes
|
||||
budget()
|
||||
if depth > 40 or len(seen) + len(indirect) > MAX_OBJECTS:
|
||||
raise ValueError("PDF object budget exceeded")
|
||||
if isinstance(value, IndirectObject):
|
||||
identity = (value.idnum, value.generation)
|
||||
if identity in indirect: return
|
||||
indirect.add(identity)
|
||||
walk(value.get_object(), depth + 1)
|
||||
return
|
||||
if isinstance(value, (DictionaryObject, ArrayObject)):
|
||||
if id(value) in seen: return
|
||||
# Retain objects as well as IDs: parsed operand containers could
|
||||
# otherwise be collected and their IDs reused across streams.
|
||||
seen[id(value)] = value
|
||||
if isinstance(value, DictionaryObject):
|
||||
if any(str(key) in FORBIDDEN for key in value) or str(value.get("/Type", "")) in ("/Filespec", "/Action", "/ObjStm", "/EmbeddedFile", "/XRef"):
|
||||
raise ValueError("Unsupported active PDF content")
|
||||
if value.get("/Type") == "/Font" and (value.get("/Subtype") != "/Type1" or value.get("/BaseFont") not in STANDARD_FONTS or "/FontDescriptor" in value or "/ToUnicode" in value or value.get("/Encoding") not in (None, "/WinAnsiEncoding", "/MacRomanEncoding", "/StandardEncoding")):
|
||||
raise ValueError("Unsupported PDF font encoding")
|
||||
if isinstance(value, StreamObject):
|
||||
if id(value) not in allowed_streams:
|
||||
raise ValueError("Unsupported hidden PDF stream")
|
||||
decoded = value.get_data()
|
||||
extracted_bytes += len(decoded)
|
||||
if extracted_bytes > MAX_BYTES:
|
||||
raise ValueError("PDF expansion budget exceeded")
|
||||
try: append(decoded.decode("utf-8", errors="strict"))
|
||||
except UnicodeError: pass
|
||||
if value.get("/Subtype") != "/Image":
|
||||
operations = ContentStream(value, reader).operations
|
||||
if len(operations) > MAX_OBJECTS: raise ValueError("PDF operation budget exceeded")
|
||||
for operands, _operator in operations:
|
||||
for operand in operands: walk(operand, depth + 1)
|
||||
for key, item in value.items():
|
||||
append(str(key))
|
||||
walk(item, depth + 1)
|
||||
elif isinstance(value, ArrayObject):
|
||||
for item in value: walk(item, depth + 1)
|
||||
elif isinstance(value, (TextStringObject, NameObject)): append(str(value))
|
||||
elif isinstance(value, ByteStringObject):
|
||||
try: append(bytes(value).decode("utf-8", errors="strict"))
|
||||
except UnicodeError: raise ValueError("Unsupported PDF string encoding")
|
||||
with stage("pdf.structure"):
|
||||
reader = PdfReader(io.BytesIO(data), strict=True)
|
||||
if reader.is_encrypted or reader.trailer.get("/Prev") or not 0 < len(reader.pages) <= MAX_PAGES:
|
||||
raise ValueError("Unsupported PDF document")
|
||||
# Only page content and page-reachable image resources may carry streams.
|
||||
# Fonts/alternate encodings, hidden orphan streams and XMP streams remain
|
||||
# outside this deliberately explicit static PDF subset.
|
||||
def allow_resource(value, depth=0, visited=None):
|
||||
if visited is None: visited = set()
|
||||
budget()
|
||||
if depth > 40 or len(visited) > MAX_OBJECTS: raise ValueError("PDF resource budget exceeded")
|
||||
if isinstance(value, IndirectObject): value = value.get_object()
|
||||
if isinstance(value, (DictionaryObject, ArrayObject)):
|
||||
if id(value) in visited: return
|
||||
visited.add(id(value))
|
||||
if isinstance(value, StreamObject):
|
||||
if value.get("/Subtype") not in ("/Image", "/Form"):
|
||||
raise ValueError("Unsupported PDF resource stream")
|
||||
allowed_streams.add(id(value))
|
||||
if isinstance(value, DictionaryObject):
|
||||
for item in value.values(): allow_resource(item, depth + 1, visited)
|
||||
elif isinstance(value, ArrayObject):
|
||||
for item in value: allow_resource(item, depth + 1, visited)
|
||||
for page in reader.pages:
|
||||
contents = page.get("/Contents")
|
||||
if isinstance(contents, IndirectObject): contents = contents.get_object()
|
||||
if isinstance(contents, ArrayObject):
|
||||
for item in contents:
|
||||
item = item.get_object()
|
||||
if not isinstance(item, StreamObject): raise ValueError("Unsupported PDF page content")
|
||||
allowed_streams.add(id(item))
|
||||
elif isinstance(contents, StreamObject): allowed_streams.add(id(contents))
|
||||
elif contents is not None: raise ValueError("Unsupported PDF page content")
|
||||
if page.get("/Resources") is not None: allow_resource(page["/Resources"])
|
||||
walk(reader.trailer)
|
||||
# Inspect every indexed object, including unreferenced metadata/actions.
|
||||
for generation, entries in reader.xref.items():
|
||||
for number in entries:
|
||||
if number: walk(IndirectObject(number, generation, reader))
|
||||
for number in reader.xref_objStm:
|
||||
walk(IndirectObject(number, 0, reader))
|
||||
media_bytes, pixels = 0, 0
|
||||
with pdfium.PdfDocument(data) as document:
|
||||
if len(document) != len(reader.pages): raise ValueError("PDF parser disagreement")
|
||||
for index, source in enumerate(reader.pages):
|
||||
budget()
|
||||
append(source.extract_text() or "")
|
||||
# Hidden/unpainted image resources also require independent OCR.
|
||||
with stage("pdf.images"):
|
||||
for image in source.images:
|
||||
if len(image.data) > MAX_BYTES: raise ValueError("PDF image budget exceeded")
|
||||
media_bytes += len(image.data)
|
||||
if media_bytes > MAX_BYTES or len(media) >= 64: raise ValueError("PDF media budget exceeded")
|
||||
value, complete = ocr_image(image.data)
|
||||
if not complete: raise ValueError("Incomplete PDF image OCR")
|
||||
append(value)
|
||||
media.append(("image", image.data))
|
||||
page = document[index]
|
||||
try:
|
||||
width, height = page.get_size()
|
||||
if not all(math.isfinite(value) and value > 0 for value in (width, height)):
|
||||
raise ValueError("Invalid PDF page dimensions")
|
||||
count = math.ceil(width * 2) * math.ceil(height * 2)
|
||||
pixels += count
|
||||
if count > MAX_PAGE_PIXELS or pixels > MAX_PIXELS:
|
||||
raise ValueError("PDF rendering budget exceeded")
|
||||
with stage("pdf.render"):
|
||||
bitmap = page.render(scale=2, draw_annots=False)
|
||||
try:
|
||||
rendered = io.BytesIO()
|
||||
bitmap.to_pil().save(rendered, format="PNG")
|
||||
payload = rendered.getvalue()
|
||||
finally: bitmap.close()
|
||||
media_bytes += len(payload)
|
||||
if media_bytes > MAX_BYTES or len(media) >= 64: raise ValueError("PDF rendered media budget exceeded")
|
||||
with stage("pdf.ocr"):
|
||||
value, complete = ocr_image(payload)
|
||||
if not complete: raise ValueError("Incomplete PDF rendered OCR")
|
||||
append(value)
|
||||
media.append(("image", payload))
|
||||
finally: page.close()
|
||||
budget()
|
||||
return "\n".join(text), True, media
|
||||
@@ -0,0 +1,2 @@
|
||||
# Official PyPI Windows x64 wheel, includes PDFium and its license notices.
|
||||
pypdfium2==5.14.0 --hash=sha256:149fd5c6397b8df8bf7911a93506eff0be874f877afe7ac936cf5d37d21a6a06
|
||||
@@ -1,4 +1,5 @@
|
||||
pypdf>=6,<8
|
||||
pypdfium2==5.14.0
|
||||
Pillow>=12,<14
|
||||
defusedxml>=0.7,<1
|
||||
yara-python==4.5.4
|
||||
|
||||
@@ -5,7 +5,7 @@ import math
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
|
||||
STAGES = frozenset(('inspect', 'parse', 'ocr', 'ocr.load', 'yara', 'semantic', 'semantic.load', 'macros'))
|
||||
STAGES = frozenset(('inspect', 'parse', 'ocr', 'ocr.load', 'yara', 'semantic', 'semantic.load', 'macros', 'pdf.structure', 'pdf.images', 'pdf.render', 'pdf.ocr', 'office.archive', 'office.xml', 'office.relationships', 'office.ocr'))
|
||||
_current = contextvars.ContextVar('inspection_timings', default=None)
|
||||
|
||||
def enabled(policy):
|
||||
|
||||
@@ -14,9 +14,12 @@ class InspectionTests(unittest.TestCase):
|
||||
def test_office_extraction(self):
|
||||
output=io.BytesIO()
|
||||
with zipfile.ZipFile(output,"w") as archive:
|
||||
archive.writestr("word/document.xml",'<document xmlns="urn:test"><t>Secret Roadmap</t></document>')
|
||||
archive.writestr("[Content_Types].xml",'<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types"><Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/><Override PartName="/word/document.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/></Types>')
|
||||
archive.writestr("_rels/.rels",'<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships"><Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="word/document.xml"/></Relationships>')
|
||||
archive.writestr("word/document.xml",'<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"><w:body><w:p><w:r><w:t>Secret Roadmap</w:t></w:r></w:p></w:body></w:document>')
|
||||
result=inspect_content(output.getvalue(),policy())
|
||||
self.assertEqual(result["contentType"],"office");self.assertEqual(result["ruleId"],"secret")
|
||||
self.assertTrue(result["coverage"]["complete"])
|
||||
def test_pdf_extraction(self):
|
||||
from pypdf import PdfWriter
|
||||
from pypdf.generic import DecodedStreamObject,NameObject,DictionaryObject
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
import io
|
||||
import unittest
|
||||
import zlib
|
||||
from unittest.mock import patch
|
||||
from pypdf import PdfWriter, PdfReader
|
||||
from pypdf.generic import NameObject, DictionaryObject, DecodedStreamObject, EncodedStreamObject, TextStringObject
|
||||
from engine import inspect_content
|
||||
from pdf_documents import extract_pdf
|
||||
|
||||
|
||||
def policy():
|
||||
return {"dlp": {"maxBytes": 1048576, "rules": [{"id": "secret", "kind": "literal", "value": "Secret Roadmap"}], "semantic": {"enabled": False}}}
|
||||
|
||||
|
||||
def pdf(value=b'BT /F1 24 Tf 20 150 Td (Public information) Tj ET', mutate=None):
|
||||
writer = PdfWriter()
|
||||
page = writer.add_blank_page(400, 220)
|
||||
font = DictionaryObject({NameObject('/Type'): NameObject('/Font'), NameObject('/Subtype'): NameObject('/Type1'), NameObject('/BaseFont'): NameObject('/Helvetica')})
|
||||
page[NameObject('/Resources')] = DictionaryObject({NameObject('/Font'): DictionaryObject({NameObject('/F1'): writer._add_object(font)})})
|
||||
stream = DecodedStreamObject(); stream.set_data(value)
|
||||
page[NameObject('/Contents')] = writer._add_object(stream)
|
||||
if mutate: mutate(writer, page)
|
||||
output = io.BytesIO(); writer.write(output)
|
||||
return output.getvalue()
|
||||
|
||||
|
||||
class PdfTests(unittest.TestCase):
|
||||
def test_public_render_and_ocr_all_pages(self):
|
||||
result = inspect_content(pdf(), policy())
|
||||
self.assertEqual(result['decision'], 'allow')
|
||||
self.assertTrue(result['coverage']['complete'])
|
||||
self.assertEqual(result['coverage']['mediaItems'], 1)
|
||||
|
||||
def test_encoded_secret_decoded_text(self):
|
||||
data = pdf(b'BT /F1 24 Tf 20 150 Td <53656372657420526F61646D6170> Tj ET')
|
||||
self.assertNotIn(b'Secret Roadmap', data)
|
||||
self.assertEqual(inspect_content(data, policy())['ruleId'], 'secret')
|
||||
|
||||
def test_hidden_metadata_and_unreferenced_strings(self):
|
||||
for mutation in [lambda w, _: w.add_metadata({'/Title': 'Secret Roadmap'}), lambda w, _: w._add_object(TextStringObject('Secret Roadmap'))]:
|
||||
self.assertEqual(inspect_content(pdf(mutate=mutation), policy())['ruleId'], 'secret')
|
||||
|
||||
def test_render_only_vector_secret_has_no_extractable_text(self):
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
mask = Image.new('L', (500, 80), 255)
|
||||
ImageDraw.Draw(mask).text((10, 8), 'Secret Roadmap', font=ImageFont.load_default(size=42), fill=0)
|
||||
operations = []
|
||||
for y in range(mask.height):
|
||||
x = 0
|
||||
while x < mask.width:
|
||||
if mask.getpixel((x, y)) < 128:
|
||||
start = x
|
||||
while x < mask.width and mask.getpixel((x, y)) < 128: x += 1
|
||||
operations.append(f'{start + 15} {180-y} {x-start} 1 re f')
|
||||
else: x += 1
|
||||
data = pdf(('\n'.join(operations)).encode())
|
||||
self.assertFalse(PdfReader(io.BytesIO(data)).pages[0].extract_text())
|
||||
self.assertNotIn(b'Secret Roadmap', data)
|
||||
self.assertEqual(inspect_content(data, policy())['ruleId'], 'secret')
|
||||
|
||||
def test_embedded_raster_secret(self):
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
image = Image.new('RGB', (700, 200), 'white')
|
||||
ImageDraw.Draw(image).text((20, 40), 'Secret Roadmap', font=ImageFont.load_default(size=48), fill='black')
|
||||
output = io.BytesIO(); image.save(output, format='PDF')
|
||||
self.assertEqual(inspect_content(output.getvalue(), policy())['ruleId'], 'secret')
|
||||
|
||||
def test_unsupported_active_encrypted_or_hidden_stream(self):
|
||||
def action(writer, _): writer._root_object[NameObject('/OpenAction')] = DictionaryObject({NameObject('/S'): NameObject('/JavaScript')})
|
||||
def attachment(writer, _): writer.add_attachment('private.txt', b'opaque')
|
||||
def encryption(writer, _): writer.encrypt('private')
|
||||
def hidden_stream(writer, _):
|
||||
stream = DecodedStreamObject(); stream.set_data(b'opaque'); writer._add_object(stream)
|
||||
for mutation in [action, attachment, encryption, hidden_stream]:
|
||||
result = inspect_content(pdf(mutate=mutation), policy())
|
||||
self.assertEqual(result['decision'], 'block')
|
||||
self.assertFalse(result['coverage']['complete'])
|
||||
|
||||
def test_decoder_and_render_budgets_fail_before_ocr(self):
|
||||
def bomb(writer, page):
|
||||
stream = EncodedStreamObject(); stream._data = zlib.compress(b'x' * (21 * 1024 * 1024))
|
||||
stream[NameObject('/Filter')] = NameObject('/FlateDecode')
|
||||
reference = page.raw_get('/Contents')
|
||||
stream.indirect_reference = reference
|
||||
writer._objects[reference.idnum - 1] = stream
|
||||
from pypdf.filters import FlateDecode
|
||||
from pypdf import get_configuration
|
||||
original_decode = FlateDecode.decode
|
||||
decodes = []
|
||||
def bounded_decode(*args, **kwargs):
|
||||
decodes.append(get_configuration().zlib_maximum_output_length)
|
||||
return original_decode(*args, **kwargs)
|
||||
with patch('engine.ocr_image', side_effect=AssertionError('Must not OCR oversized input')):
|
||||
with patch.object(FlateDecode, 'decode', side_effect=bounded_decode):
|
||||
self.assertEqual(inspect_content(pdf(mutate=bomb), policy())['decision'], 'block')
|
||||
self.assertEqual(decodes, [20 * 1024 * 1024])
|
||||
self.assertEqual(inspect_content(pdf(mutate=lambda _w, page: setattr(page.mediabox, 'right', 1000000)), policy())['decision'], 'block')
|
||||
|
||||
def test_missing_ocr_never_claims_complete(self):
|
||||
with self.assertRaises(ValueError): extract_pdf(pdf(), lambda _data: ('', False))
|
||||
|
||||
def test_debug_stages_record_actual_structure_render_and_ocr(self):
|
||||
from stage_timings import collect, metadata, validated
|
||||
p = policy(); p['diagnostics'] = {'enabled': True, 'mode': 'debug'}
|
||||
with collect(p) as state:
|
||||
result = inspect_content(pdf(), p)
|
||||
self.assertEqual(result['decision'], 'allow')
|
||||
envelope = validated(metadata(state))
|
||||
self.assertIsNotNone(envelope)
|
||||
stages = {record['stage'] for record in envelope['records']}
|
||||
self.assertTrue({'pdf.structure', 'pdf.images', 'pdf.render', 'pdf.ocr'} <= stages)
|
||||
for record in envelope['records']:
|
||||
self.assertGreaterEqual(record['durationMs'], 0)
|
||||
if record['stage'].startswith('pdf.'): self.assertIsNotNone(record['parentId'])
|
||||
|
||||
|
||||
if __name__ == '__main__': unittest.main()
|
||||
@@ -2,7 +2,7 @@
|
||||
function validateStages(envelope,now=Date.now()){try{
|
||||
const own=(object,key)=>{const d=Object.getOwnPropertyDescriptor(object,key);return d&&'value'in d?d.value:undefined;};
|
||||
if(!envelope||typeof envelope!=='object')return null;const base=own(envelope,'baseUnixMs'),records=own(envelope,'records');if(!Number.isSafeInteger(base)||base<now-60000||base>now+2000||!Array.isArray(records)||records.length>64)return null;
|
||||
const allowed=new Set(['inspect','parse','ocr','yara','semantic','macros','ocr.load','semantic.load']),seen=new Map(),validated=[];
|
||||
const allowed=new Set(['inspect','parse','ocr','yara','semantic','macros','ocr.load','semantic.load','pdf.structure','pdf.images','pdf.render','pdf.ocr','office.archive','office.xml','office.relationships','office.ocr']),seen=new Map(),validated=[];
|
||||
const identifier=value=>Number.isSafeInteger(value)&&value>=0||typeof value==='string'&&/^[A-Za-z0-9_-]{1,64}$/.test(value);
|
||||
for(let i=0;i<records.length;i++){const record=own(records,String(i));if(!record||typeof record!=='object')return null;const id=own(record,'id'),parentId=own(record,'parentId'),stage=own(record,'stage'),start=own(record,'startOffsetMs'),duration=own(record,'durationMs'),status=own(record,'status');if(!identifier(id)||seen.has(id)||parentId!==null&&!seen.has(parentId)||!allowed.has(stage)||!Number.isFinite(start)||!Number.isFinite(duration)||start<0||duration<0||start+duration>30000||base+start+duration>now+2000||!['ok','error'].includes(status))return null;
|
||||
const item={id,parentId,stage,start,duration,status};seen.set(id,item);validated.push(item);
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Build a pinned private Windows x64 inspector runtime; no machine-wide install."""
|
||||
import hashlib
|
||||
import argparse
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
@@ -9,6 +10,9 @@ import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
if sys.platform!='win32': raise SystemExit('Windows runtime provisioning runs on Windows only')
|
||||
parser=argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--verify-existing',action='store_true',help='Verify exact installed locks/wheel before updating manifest; no dependency reinstall')
|
||||
arguments=parser.parse_args()
|
||||
VERSION='3.13.16'
|
||||
CHECKSUM='97dae5274cc54867065e8d5a3226e48c35017ed332a0fdb0e27d5b5821961297'
|
||||
root=Path.cwd().resolve()
|
||||
@@ -16,11 +20,26 @@ directory=root/'.runtime'/'windows'/'python'
|
||||
directory.mkdir(parents=True,exist_ok=True)
|
||||
requirements=root/'apps'/'inspector'/'requirements.lock.txt'
|
||||
macro_requirements=root/'apps'/'inspector'/'requirements-macros.lock.txt'
|
||||
requirements_hash=hashlib.sha256(requirements.read_bytes()+macro_requirements.read_bytes()).hexdigest()
|
||||
pdf_requirements=root/'apps'/'inspector'/'requirements-pdf.lock.txt'
|
||||
requirements_hash=hashlib.sha256(requirements.read_bytes()+macro_requirements.read_bytes()+pdf_requirements.read_bytes()).hexdigest()
|
||||
manifest=directory/'safetynet-runtime.json'
|
||||
expected={'python':VERSION,'architecture':'amd64','sha256':CHECKSUM,'requirementsHash':requirements_hash}
|
||||
previous=json.loads(manifest.read_text()) if manifest.exists() else {}
|
||||
if previous!=expected:
|
||||
pdf_wheel=directory.parent/'pypdfium2-5.14.0-py3-none-win_amd64.whl'
|
||||
pdf_checksum='149fd5c6397b8df8bf7911a93506eff0be874f877afe7ac936cf5d37d21a6a06'
|
||||
if not arguments.verify_existing and not pdf_wheel.exists():
|
||||
# Retain the reviewed wheel for independent verification on fresh builds.
|
||||
# Verification mode remains offline and never repairs a missing archive.
|
||||
subprocess.run([sys.executable,'-m','pip','download','--dest',str(directory.parent),
|
||||
'--platform','win_amd64','--python-version','3.13','--implementation','cp','--abi','cp313',
|
||||
'--only-binary=:all:','--no-deps','--require-hashes','-r',str(pdf_requirements)],check=True)
|
||||
if not pdf_wheel.is_file() or hashlib.sha256(pdf_wheel.read_bytes()).hexdigest()!=pdf_checksum:
|
||||
raise SystemExit('Retained PDF renderer wheel missing or checksum mismatch')
|
||||
if arguments.verify_existing:
|
||||
if any(previous.get(k)!=expected[k] for k in ('python','architecture','sha256')):
|
||||
raise SystemExit('Existing private runtime identity mismatch')
|
||||
subprocess.run([str(directory/'python.exe'),str(root/'scripts/verify-python-dependencies.py')],check=True)
|
||||
if previous!=expected and not arguments.verify_existing:
|
||||
archive=directory.parent/f'python-{VERSION}-embed-amd64.zip'
|
||||
if not archive.exists() or hashlib.sha256(archive.read_bytes()).hexdigest()!=CHECKSUM:
|
||||
with urllib.request.urlopen(f'https://www.python.org/ftp/python/{VERSION}/python-{VERSION}-embed-amd64.zip',timeout=60) as response:
|
||||
@@ -42,10 +61,15 @@ if previous!=expected:
|
||||
subprocess.run([sys.executable,'-m','pip','install','--target',str(directory/'Lib'/'site-packages'),
|
||||
'--platform','win_amd64','--python-version','3.13','--implementation','cp','--abi','cp313',
|
||||
'--only-binary=:all:','--no-deps','--require-hashes','--upgrade','-r',str(macro_requirements)],check=True)
|
||||
subprocess.run([sys.executable,'-m','pip','install','--target',str(directory/'Lib'/'site-packages'),
|
||||
'--platform','win_amd64','--python-version','3.13','--implementation','cp','--abi','cp313',
|
||||
'--only-binary=:all:','--no-deps','--require-hashes','--upgrade','-r',str(pdf_requirements)],check=True)
|
||||
manifest.write_text(json.dumps(expected,indent=2))
|
||||
inspector=directory.parent/'inspector'
|
||||
inspector.mkdir(exist_ok=True)
|
||||
for name in ['stage_timings.py','office_macros.py','engine.py','server.py','semantic.py','windows_containment.py','windows_appcontainer.py','sandbox_worker.py','provision_runtime_acl.py']:
|
||||
for name in ['stage_timings.py','pdf_documents.py','office_content.py','office_macros.py','engine.py','server.py','semantic.py','windows_containment.py','windows_appcontainer.py','sandbox_worker.py','provision_runtime_acl.py']:
|
||||
shutil.copyfile(root/'apps'/'inspector'/name,inspector/name)
|
||||
subprocess.run([str(directory/'python.exe'),'-c','import pypdf, PIL, defusedxml, yara, engine; assert engine.inspect_content(b"public", {"dlp":{"maxBytes":1024,"rules":[],"semantic":{"enabled":False}}})["decision"] == "allow"; assert yara.compile(source="rule test { condition: true }").match(data=b"test"); engine.local_ocr(); print("Bundled Windows inspector, YARA and offline ONNX OCR runtime verified.")'],check=True)
|
||||
subprocess.run([str(directory/'python.exe'),str(root/'apps/inspector/provision_runtime_acl.py'),'--provision'],check=True)
|
||||
if arguments.verify_existing:
|
||||
manifest.write_text(json.dumps(expected,indent=2))
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Read-only verification of this exact private runtime's pinned distributions.
|
||||
|
||||
Installed RECORD checks detect inconsistent installation; only the PDFium wheel
|
||||
comparison below supplies independent pinned artifact provenance. Neither is an
|
||||
independent attestation of all base package wheels beyond prior provisioning.
|
||||
"""
|
||||
import base64
|
||||
import hashlib
|
||||
import importlib.metadata
|
||||
import re
|
||||
import sys
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
runtime = root / '.runtime/windows/python'
|
||||
site = runtime / 'Lib/site-packages'
|
||||
if Path(sys.executable).resolve().parent != runtime.resolve():
|
||||
raise SystemExit('Use the exact private Windows Python runtime')
|
||||
|
||||
names = []
|
||||
checked = 0
|
||||
def file_hash(path):
|
||||
digest = hashlib.sha256()
|
||||
with path.open('rb') as file:
|
||||
for chunk in iter(lambda: file.read(1048576), b''): digest.update(chunk)
|
||||
return digest.digest()
|
||||
|
||||
for filename in ('requirements.lock.txt', 'requirements-macros.lock.txt', 'requirements-pdf.lock.txt'):
|
||||
for line in (root / 'apps/inspector' / filename).read_text(encoding='utf-8').splitlines():
|
||||
if not line or line.startswith('#'): continue
|
||||
match = re.match(r'^([A-Za-z0-9_.-]+)==([^\s]+)\s+--hash=sha256:', line)
|
||||
if not match: raise SystemExit('Runtime dependency lock rejected')
|
||||
name, version = match.groups()
|
||||
distribution = importlib.metadata.distribution(name)
|
||||
if distribution.version != version: raise SystemExit('Runtime dependency version mismatch')
|
||||
names.append(name)
|
||||
for entry in distribution.files or []:
|
||||
if entry.hash is None: continue
|
||||
path = distribution.locate_file(entry).resolve()
|
||||
if not path.is_relative_to(runtime.resolve()) or not path.is_file(): raise SystemExit('Runtime dependency file missing')
|
||||
if entry.hash.mode != 'sha256': raise SystemExit('Runtime dependency hash algorithm rejected')
|
||||
actual = base64.urlsafe_b64encode(file_hash(path)).decode().rstrip('=')
|
||||
if actual != entry.hash.value: raise SystemExit('Runtime dependency file integrity mismatch')
|
||||
checked += 1
|
||||
|
||||
wheel = root / '.runtime/windows/pypdfium2-5.14.0-py3-none-win_amd64.whl'
|
||||
expected = '149fd5c6397b8df8bf7911a93506eff0be874f877afe7ac936cf5d37d21a6a06'
|
||||
if not wheel.is_file() or hashlib.sha256(wheel.read_bytes()).hexdigest() != expected:
|
||||
raise SystemExit('Pinned PDFium wheel verification failed')
|
||||
pdf_files = 0
|
||||
with zipfile.ZipFile(wheel) as archive:
|
||||
for entry in archive.infolist():
|
||||
if entry.is_dir() or entry.filename.endswith('.dist-info/RECORD'): continue
|
||||
target = (site / entry.filename).resolve()
|
||||
if not target.is_relative_to(site.resolve()) or not target.is_file() or target.read_bytes() != archive.read(entry):
|
||||
raise SystemExit('Installed PDFium differs from pinned wheel')
|
||||
pdf_files += 1
|
||||
import pypdfium2
|
||||
if pypdfium2.PYPDFIUM_INFO.api_tag != (5, 14, 0) or pypdfium2.PDFIUM_INFO.flags:
|
||||
raise SystemExit('PDFium feature manifest rejected')
|
||||
licenses = site / 'pypdfium2-5.14.0.dist-info/licenses'
|
||||
if not (licenses / 'LICENSES/Apache-2.0.txt').is_file() or not (licenses / 'LICENSES/BSD-3-Clause.txt').is_file() or not any((licenses / 'data/windows_x64/BUILD_LICENSES').glob('*')):
|
||||
raise SystemExit('PDFium license notices missing')
|
||||
print(f'Private runtime verified: {len(names)} pinned distributions, {checked} RECORD hashes, {pdf_files} pinned PDFium wheel files; license notices retained.')
|
||||
@@ -0,0 +1,114 @@
|
||||
"""Real OPC ZIP fixtures for strict static OOXML coverage and hostile parts."""
|
||||
import base64
|
||||
import io
|
||||
import json
|
||||
import sys
|
||||
import unittest
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
sys.path.insert(0,str(Path(__file__).resolve().parent.parent/'apps/inspector'))
|
||||
from stage_timings import collect,stage,metadata,validated
|
||||
from office_content import extract_office,OfficeCoverageError,CT,REL,R,W,S,P,A,CP,MAIN,TYPE_PREFIX
|
||||
PNG=base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAusB9Wl6b5sAAAAASUVORK5CYII=')
|
||||
def xml(tag,namespace,inside=''):return f'<{tag} xmlns="{namespace}" xmlns:r="{R}">{inside}</{tag}>'.encode()
|
||||
def relations(items):return xml('Relationships',REL,''.join(f'<Relationship Id="{id_}" Type="{type_}" Target="{target}"'+(f' TargetMode="{mode}"' if mode else '')+'/>' for id_,type_,target,mode in items))
|
||||
def package(family='docx',parts=None,main=None,relationships=None,extra_info=None):
|
||||
main_name={'docx':'word/document.xml','xlsx':'xl/workbook.xml','pptx':'ppt/presentation.xml'}[family]
|
||||
main_type=next(type_ for type_,values in MAIN.items() if values[0]==family)
|
||||
default={'docx':xml('document',W,'<body><p><r><t>Public company newsletter</t></r></p></body>'),'xlsx':xml('workbook',S,'<sheets><sheet name="Public budget" sheetId="1" r:id="rId1"/></sheets>'),'pptx':xml('presentation',P,'<sldIdLst><sldId id="256" r:id="rId1"/></sldIdLst>')}
|
||||
values={main_name:(main_type,main or default[family])};values.update(parts or {})
|
||||
if family=='xlsx' and 'xl/worksheets/sheet1.xml' not in values:values['xl/worksheets/sheet1.xml']=(TYPE_PREFIX+'spreadsheetml.worksheet+xml',xml('worksheet',S,'<sheetData><row r="1"><c r="A1" t="inlineStr"><is><t>Public projected revenue</t></is></c></row></sheetData>'))
|
||||
if family=='pptx' and 'ppt/slides/slide1.xml' not in values:values['ppt/slides/slide1.xml']=(TYPE_PREFIX+'presentationml.slide+xml',xml('sld',P,f'<cSld><spTree><nvGrpSpPr/><grpSpPr/><sp><nvSpPr/><spPr/><txBody xmlns:a="{A}"><a:bodyPr/><a:lstStyle/><a:p><a:r><a:t>Public quarterly update</a:t></a:r></a:p></txBody></sp></spTree></cSld>'))
|
||||
rels={'_rels/.rels':[('root',R+'/officeDocument',main_name,None)]}
|
||||
if family=='xlsx':rels['xl/_rels/workbook.xml.rels']=[('rId1',R+'/worksheet','worksheets/sheet1.xml',None)]
|
||||
if family=='pptx':rels['ppt/_rels/presentation.xml.rels']=[('rId1',R+'/slide','slides/slide1.xml',None)]
|
||||
rels.update(relationships or {})
|
||||
types=f'<Types xmlns="{CT}"><Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>'+''.join(f'<Override PartName="/{name}" ContentType="{type_}"/>' for name,(type_,_payload) in values.items())+'</Types>'
|
||||
output=io.BytesIO()
|
||||
with zipfile.ZipFile(output,'w',zipfile.ZIP_DEFLATED) as archive:
|
||||
archive.writestr('[Content_Types].xml',types)
|
||||
for name,(_type,payload) in values.items():archive.writestr(name,payload)
|
||||
for name,items in rels.items():archive.writestr(name,relations(items))
|
||||
if extra_info:archive.writestr(extra_info,b'PUBLIC')
|
||||
return output.getvalue()
|
||||
def extract(data,callback=lambda _data:('',True),**limits):return extract_office(data,callback,**limits)
|
||||
class OfficeTests(unittest.TestCase):
|
||||
def test_public_valid_docx_xlsx_and_pptx_static_parts_complete(self):
|
||||
for family in ['docx','xlsx','pptx']:
|
||||
result=extract(package(family));self.assertTrue(result['complete'],result['reasons']);self.assertEqual(result['family'],family);self.assertIn('Public',result['text'])
|
||||
def test_core_metadata_attributes_namespace_and_split_runs_included(self):
|
||||
main=xml('document',W,'<body><p secret="ATTR-SECRET"><r><t>TOP</t></r><r><t>SECRET</t></r></p></body>')
|
||||
metadata=xml('coreProperties',CP,'<title xmlns="http://purl.org/dc/elements/1.1/">METADATA-SECRET</title>')
|
||||
result=extract(package(parts={'docProps/core.xml':('application/vnd.openxmlformats-package.core-properties+xml',metadata)},main=main))
|
||||
self.assertTrue(result['complete']);self.assertIn('ATTR-SECRET',result['text']);self.assertIn('METADATA-SECRET',result['text']);self.assertIn('TOPSECRET',result['text']);self.assertIn(W,result['text'])
|
||||
def test_shared_strings_and_hidden_unreferenced_parts_are_inspected(self):
|
||||
parts={'xl/sharedStrings.xml':(TYPE_PREFIX+'spreadsheetml.sharedStrings+xml',xml('sst',S,'<si><t>SHARED-SECRET</t></si>')),'xl/worksheets/hidden.xml':(TYPE_PREFIX+'spreadsheetml.worksheet+xml',xml('worksheet',S,'<sheetData><row><c t="inlineStr"><is><t>HIDDEN-SECRET</t></is></c></row></sheetData>'))}
|
||||
result=extract(package('xlsx',parts));self.assertTrue(result['complete']);self.assertIn('SHARED-SECRET',result['text']);self.assertIn('HIDDEN-SECRET',result['text'])
|
||||
def test_image_text_receives_ocr_and_cannot_escape_dlp_input(self):
|
||||
parts={'word/media/image1.png':('image/png',PNG)};calls=[]
|
||||
def ocr(payload):calls.append(payload);return ('RASTER-SECRET',True)
|
||||
result=extract(package(parts=parts),ocr);self.assertTrue(result['complete']);self.assertEqual(calls,[PNG]);self.assertIn('RASTER-SECRET',result['text']);self.assertEqual(len(result['media']),1)
|
||||
self.assertFalse(extract(package(parts=parts),lambda _data:('',False))['complete'])
|
||||
def test_external_relationship_metadata_scanned_but_coverage_incomplete(self):
|
||||
result=extract(package(relationships={'word/_rels/document.xml.rels':[('external',R+'/hyperlink','https://host/RELATIONSHIP-SECRET','External')]}))
|
||||
self.assertFalse(result['complete']);self.assertIn('RELATIONSHIP-SECRET',result['text'])
|
||||
def test_unsupported_font_ole_macro_equation_chart_and_vector_fail_closed(self):
|
||||
for name,type_,payload in [('word/fonts/private.odttf','application/vnd.openxmlformats-officedocument.obfuscatedFont',b'font'),('word/embeddings/a.bin','application/vnd.openxmlformats-officedocument.oleObject',b'OLE'),('word/vbaProject.bin','application/vnd.ms-office.vbaProject',b'VBA'),('word/chart.xml',TYPE_PREFIX+'drawingml.chart+xml',xml('chartSpace',A,'<chart/>'))]:
|
||||
self.assertFalse(extract(package(parts={name:(type_,payload)}))['complete'])
|
||||
for content in ['<oMath xmlns="http://schemas.openxmlformats.org/officeDocument/2006/math"/>',f'<custGeom xmlns="{A}"/>','<fldSimple instr="DDE secret"/>']:
|
||||
self.assertFalse(extract(package(main=xml('document',W,'<body>'+content+'</body>')))['complete'])
|
||||
def test_unknown_xml_content_type_is_incomplete_even_if_text_readable(self):
|
||||
result=extract(package(parts={'word/private.xml':('application/x-private+xml',xml('document',W,'<t>SECRET</t>'))}));self.assertFalse(result['complete']);self.assertIn('SECRET',result['text'])
|
||||
def test_duplicate_case_paths_traversal_and_compression_bombs_rejected(self):
|
||||
for extra in ['word/../outside','WORD/document.xml']:
|
||||
with self.assertRaises(OfficeCoverageError):extract(package(extra_info=extra))
|
||||
with self.assertRaises(OfficeCoverageError):extract(package(main=xml('document',W,'A'*100000)))
|
||||
def test_relationship_target_missing_escape_id_duplicate_and_role_mismatch_rejected(self):
|
||||
for items in [[('x',R+'/styles','missing.xml',None)],[('x',R+'/styles','../../../escape.xml',None)],[('x',R+'/styles','document.xml',None)],[('x',R+'/officeDocument','document.xml',None),('x',R+'/officeDocument','document.xml',None)]]:
|
||||
with self.assertRaises(OfficeCoverageError):extract(package(relationships={'word/_rels/document.xml.rels':items}))
|
||||
def test_dtd_and_bad_xml_are_rejected(self):
|
||||
for payload in [b'<!DOCTYPE x [<!ENTITY e "SECRET">]><x>&e;</x>',b'<x>']:
|
||||
with self.assertRaises(Exception):extract(package(main=payload))
|
||||
def test_comments_and_processing_instructions_are_not_silently_discarded(self):
|
||||
result=extract(package(main=xml('document',W,'<!--COMMENT-SECRET--><body><p><r><t>Public</t></r></p></body>')))
|
||||
self.assertTrue(result['complete']);self.assertIn('COMMENT-SECRET',result['text'])
|
||||
result=extract(package(main=xml('document',W,'<?xml-stylesheet href="PI-SECRET"?><body/>')))
|
||||
self.assertFalse(result['complete']);self.assertIn('PI-SECRET',result['text'])
|
||||
def test_simple_formula_cache_covered_dynamic_dde_and_missing_cache_incomplete(self):
|
||||
def sheet(formula,cache):return xml('worksheet',S,'<sheetData><row><c r="A1"><f>'+formula+'</f><v>'+cache+'</v></c></row></sheetData>')
|
||||
for formula,cache,complete in [('SUM(1,2)','3',True),('IF(1,"CACHE-SECRET","Public")','CACHE-SECRET',True),('CHAR(83)&CHAR(69)','SE',False),('WEBSERVICE("https://host")','Public',False),("cmd|'/c calc'!A0",'0',False),('SUM(1,2)','',False)]:
|
||||
result=extract(package('xlsx',{'xl/worksheets/sheet1.xml':(TYPE_PREFIX+'spreadsheetml.worksheet+xml',sheet(formula,cache))}))
|
||||
self.assertEqual(result['complete'],complete,formula)
|
||||
if cache:self.assertIn(cache,result['text'])
|
||||
def test_trailing_data_and_local_extra_fields_cannot_claim_coverage(self):
|
||||
with self.assertRaises(OfficeCoverageError):extract(package()+b'TRAILING-SECRET')
|
||||
info=zipfile.ZipInfo('word/extra.bin');info.extra=b'\x01\x00\x03\x00ABC'
|
||||
with self.assertRaises(OfficeCoverageError):extract(package(extra_info=info))
|
||||
def test_aggregate_stages_have_real_parent_timing_and_no_document_content(self):
|
||||
parts={'word/header'+str(index)+'.xml':(TYPE_PREFIX+'wordprocessingml.header+xml',xml('hdr',W,'<p><r><t>SYNTHETIC-PRIVATE-CONTENT</t></r></p>')) for index in range(90)}
|
||||
with collect({'diagnostics':{'enabled':True,'mode':'debug'}}) as state:
|
||||
with stage('parse'):result=extract(package(parts=parts))
|
||||
self.assertTrue(result['complete']);proof=metadata(state);self.assertIs(validated(proof),proof)
|
||||
self.assertEqual([record['stage'] for record in proof['records']],['parse','office.archive','office.xml','office.relationships','office.ocr'])
|
||||
parent=proof['records'][0]
|
||||
for record in proof['records'][1:]:
|
||||
self.assertEqual(record['parentId'],parent['id']);self.assertGreaterEqual(record['startOffsetMs'],parent['startOffsetMs']);self.assertLessEqual(record['startOffsetMs']+record['durationMs'],parent['startOffsetMs']+parent['durationMs']);self.assertEqual(record['status'],'ok')
|
||||
self.assertNotIn('SYNTHETIC-PRIVATE-CONTENT',json.dumps(proof));self.assertNotIn('header90.xml',json.dumps(proof))
|
||||
def test_error_stages_preserve_correlation_without_exception_text(self):
|
||||
def bad_ocr(_payload):raise ValueError('SYNTHETIC-PRIVATE-ERROR')
|
||||
cases=[(package()+b'TAIL',None,'office.archive'),(package(main=b'<BROKEN>'),None,'office.xml'),(package(relationships={'word/_rels/document.xml.rels':[('x',R+'/styles','missing.xml',None)]}),None,'office.relationships'),(package(parts={'word/media/image.png':('image/png',PNG)}),bad_ocr,'office.ocr')]
|
||||
for data,callback,expected in cases:
|
||||
with collect({'diagnostics':{'enabled':True,'mode':'debug'}}) as state:
|
||||
with self.assertRaises(Exception):
|
||||
with stage('parse'):extract(data,callback or (lambda _image:('',True)))
|
||||
proof=metadata(state);self.assertIs(validated(proof),proof);self.assertEqual(proof['records'][0]['status'],'error')
|
||||
record=next(record for record in proof['records'] if record['stage']==expected);self.assertEqual(record['status'],'error');self.assertEqual(record['parentId'],1);self.assertNotIn('SYNTHETIC-PRIVATE-ERROR',json.dumps(proof))
|
||||
def test_disabled_or_operational_diagnostics_do_not_change_extraction(self):
|
||||
for policy in [{},{'diagnostics':{'enabled':False,'mode':'debug'}},{'diagnostics':{'enabled':True,'mode':'operational'}}]:
|
||||
with collect(policy) as state:result=extract(package())
|
||||
self.assertTrue(result['complete']);self.assertIsNone(metadata(state))
|
||||
def test_bounds_cannot_return_complete(self):
|
||||
data=package()
|
||||
for limits in [{'max_bytes':len(data)-1},{'max_parts':1},{'max_text':10}]:
|
||||
with self.assertRaises(OfficeCoverageError):extract(data,**limits)
|
||||
if __name__=='__main__':unittest.main()
|
||||
Reference in new issue
Block a user