AI-103: Computer Vision Solutions

Image Analysis 4.0 API

• Dense Captions — up to 10 region-level captions + one overall caption
• Smart Crops — area of interest detection for thumbnail generation
• Object Detection — returns bounding boxes with labels and confidence scores
• Tag Generation — flat list of descriptive tags with confidence
• Read API (OCR) — text recognition from images and documents; returns lines and words with bounding polygons
• Face Detection — detects face bounding boxes only; recognition requires Face API (limited access)
• Background Removal / Segmentation — foreground extraction and semantic segmentation

Spatial Analysis

• People counting, zone crossing, social distancing monitoring
• Deployed via Docker container on edge or Azure
• Still appears on exam despite being deprecated in newer iterations

Video Indexer

• Shot detection — automatic chapter/scene segmentation
• Speaker diarization — who spoke when
• Transcript generation — speech-to-text across multiple languages
• OCR in video — reads on-screen text frame by frame
• Scene understanding — labels, brands, celebrities

Custom Vision

• Classification vs Object Detection: classification gives a label to the whole image; object detection returns bounding boxes
• Training datasets: need labeled images per class or annotated bounding boxes
• Evaluation metrics: Precision, Recall, Average Precision (AP), Mean Average Precision (mAP)
• Iterative training: add images, retrain, evaluate, repeat

Passing Images in Messages

{
  "role": "user",
  "content": [
    {"type": "text", "text": "Describe this chart."},
    {"type": "image_url",
     "image_url": {"url": "https://...jpg"}}
  ]
}

• Supported formats: JPEG, PNG, GIF, WEBP
• Two input methods: image URL or base64-encoded data URI
• Image URL requires the model to fetch externally; base64 keeps everything in-request

Vision Use Cases

• Document understanding (forms, invoices, reports)
• Chart reading and trend explanation
• Product inspection and defect detection
• Scene description and VQA (Visual Question Answering)
• Multimodal agent: vision + text reasoning + function calling in one loop

Prompt Engineering for Vision

• Provide detailed instructions specifying what to look for
• Request specific output formats (JSON, bullet list, table)
• Combine with system messages for role context

DALL-E 3 via Azure OpenAI

• API call: client.images.generate(...)
• prompt — text description of the desired image
• size — 1024x1024 (square), 1792x1024 (landscape), 1024x1792 (portrait)
• quality — standard (faster) or hd (higher detail)
• style — vivid (dramatic) or natural (photorealistic)
• n — number of images (DALL-E 3 supports n=1 only)
• Revised prompts: DALL-E 3 rewrites the user prompt for safety — the revised version is returned in the response
• Content policy filtering rejects unsafe prompts before generation

Model Catalog & Other Generators

• Azure AI Foundry catalog includes Stable Diffusion and other open-weight image models
• DALL-E 2 supports image editing (inpainting) and image variations — DALL-E 3 does not

• New unified service in Azure AI Foundry that consolidates document, image, audio, and video extraction
• Analyzers: pre-built analyzers for common schemas; custom analyzers for bespoke field extraction
• Replaces some Form Recognizer + Vision features in a single unified API
• Integrates with Azure AI Search for multimodal indexing and retrieval
• Field extraction from unstructured content (invoices, receipts, images, audio recordings, video clips)
• Configured and deployed inside an Azure AI Foundry project

• Image content moderation categories: Hate, Violence, Sexual, Self-Harm
• Severity levels 0–7 per category (0 = safe, 7 = most severe)
• Custom blocklists for images (hash-based matching)
• Protected material detection for images (copyright / IP)
• Groundedness check for image-based claims

• Face Detection: locate face bounding boxes; attributes (age estimate, head pose) — no approval required
• Face Verification: compare two faces, returns similarity score — limited access required
• Face Identification: match a face to a PersonGroup — limited access required (requires Microsoft approval)
• Liveness Detection: anti-spoofing (determines if the subject is a real person, not a photo) — requires limited access
• PersonGroup / FaceList: storage structures for known identities
• Emotion detection removed in newer API versions

• Multi-class classification: one label per image (mutually exclusive)
• Multi-label classification: multiple labels per image allowed
• Object detection: bounding box annotation required per object
• Quick Training vs Advanced Training: budget compute time for better accuracy
• Export formats: TensorFlow, CoreML, ONNX, Docker container
• Precision: of all predicted positives, what fraction is correct?
• Recall: of all actual positives, what fraction was found?
• mAP (mean Average Precision): average AP across all classes