I had a small, boring job to do: sort images into categories. For every image the answer is yes or no for one class, and the obvious tool for that today is a cloud call that costs a fraction of a cent. But my Mac mini now has an on-device model behind Apple Foundation Models, and a local model has no API fee at all. So I wanted to know whether it could do the job instead.

It couldn’t, at least not on its own. Apple’s model missed a third of the positives, 30 of 90. The cheap cloud option, the Decisions API with GPT-6 Luna, came close to GPT-6.1 Terra at about a tenth of the cost. A later run with the pricier GPT-6.1 Sol did no better than Terra.

Here is what I compared. Apple Foundation Models, Decisions with Luna, and Terra through the Responses API all got the same 30 image grids from COCO 2017 val. I also scored the union of the Apple and Decisions answers, to see whether the two together beat Decisions alone. About an hour after the others, I ran Sol on the same grids as a separate check. It gets its own section near the end.

The short version

Terra was the most accurate: 26 of 30 grids fully right and 265 of 270 cells. Decisions got 23 grids and 262 cells, at an estimated $0.23 per 1,000 grids against about $2.12 for Terra. With only 30 grids, the cloud setups and the union can’t be told apart. The one gap that is clearly real is Apple’s.

What this test can’t tell you

Before the details, the limits. This is 30 grids and three classes, not a general evaluation of vision models. Each setup ran once, in a fixed order. The labels come from COCO annotations, and I fixed them before any model ran. The centre crop and the area threshold also make this test different from the standard COCO detection benchmark. And COCO is an old, widely used public dataset, so the models likely saw some of these photos in training. I can’t confirm that either way.

The setups also didn’t see the photos the same way. Apple got one photo per call. Decisions got all nine questions in one HTTP request, but each question carried only its own photo. Terra got nine separate cell images in one message and could compare them, even though the prompt asked it to judge each image on its own. That design difference may explain part of the accuracy gap.

How the scores work

Each grid has nine photos and one class, such as bus, with a yes or no for every photo. A full grid match means all nine answers are right. Correct cells counts every right answer out of 270. One wrong cell fails the whole grid, so the full-grid number tells you how often a whole batch comes back right.

The results

Every row below comes from the same pass: 30 grids, 270 unique photos, 90 positive cells. Cost covers the API only.

SetupFull grids right (95% interval)Correct cellsMedian time per gridAPI cost per 1,000 grids
Answer “no” for every cell (baseline)0/30180/270n/an/a
Apple Foundation Models5/30 (7–34%)238/2705.02 s$0
Decisions API + GPT-6 Luna23/30 (59–88%)262/2701.23 s$0.23
Apple ∪ Decisions24/30 (63–91%)262/270≈6.35 s (note 1)$0.23
GPT-6.1 Terra via Responses API26/30 (70–95%)265/2702.78 s≈$2.12 (note 2)

Notes on the table:

  1. I scored the union offline from recorded answers. Its time is the median of the per-grid sums of two separate measurements, so it assumes the two setups run one after the other. A parallel pipeline would take at least as long as the slower of the two. I didn’t measure one. Decisions scored all nine cells of every grid, so the union’s API cost equals the Decisions cost.
  2. Terra’s cost uses the documented default 30m cache-write rate. The raw responses report different cache-retention metadata, which the cost section describes.
  3. The cost column extrapolates average usage from these 30 grids, without retries, taxes, or the Mac’s electricity. The figures are my calculation from token usage and published prices. No bill confirms them.

The baseline row matters more than it looks. A classifier that answers “no” for every photo never gets a full grid, because each grid has 2 to 4 positives. It still gets 180 of 270 cells right. So read the correct-cells column against 180, not against 0. Seen that way, Apple’s 238 is a lot less impressive than it first seems.

The ranges in brackets are 95% Wilson score intervals on the full-grid rate, with n = 30. I computed them per grid, not per cell, because the nine cells in a grid share a class and a request, so they aren’t independent trials. They are wide, and that is the honest picture of what 30 grids can show.

How I built the grids

The full instances_val2017.json describes the 5,000 photos of COCO 2017 val. From it, I selected 270 photos with seed 20261102. The classes were bicycle, bus, and fire hydrant, with ten grids per class. No photo repeats across grids or classes. Each grid has 2 to 4 positives, 90 in total.

A positive needed at least one non-crowd object of the target class with an annotated area of at least 1% of the original frame. After the centre crop, the visible segmentation mask of that object also had to cover at least 1% of the square. Negatives came only from photos with no annotation of the target class at all. A photo whose target object was tiny or fully cropped out never became a negative.

Each photo was cropped to its centre square, resized to 300×300 with LANCZOS, and placed into a 900×900 grid image. The inference scripts then cut the cells back out of that grid image. Every model received single cells only, never the 900×900 composite image.

Before inference, I froze the manifest, the file hashes, and a list of 57 flagged cells. A cell got a flag when its object was partly cropped, sat at the edge of the original frame, or covered 1–2% of the square. Regenerating with the same seed gave identical files. The pre-inference checks were programmatic: annotations, masks, crops, file hashes, and the response format. COCO annotations are licensed CC BY 4.0. The photos keep their own Flickr licenses, which the manifest records. See the COCO terms of use.

How each setup ran

Apple processed each cell in its own session through SystemLanguageModel.default with FoundationModels image attachments, at temperature 0 and a maximum of 256 response tokens. The Apple API doesn’t return an identifier for the system weights it used, so I didn’t record the exact model version. The macOS build number at the end of this post identifies the operating system, not the model. The prompt was:

Does this image contain any visible part of {target}? Return only {"match": true} or {"match": false}.

The cloud requests shared this text:

Offline image-recognition evaluation using COCO 2017 validation photographs. Evaluate each image independently.

Decisions used gpt-6-luna with one HTTP request per grid and nine predicate questions. Each question carried its own photo and the text Does this image contain any visible part of {target}?. The shared input held no images. A cell counted as positive at probability >= 0.5, a threshold I fixed before the test. The Decisions results depend on that threshold, and I didn’t try others.

Terra got one Responses request per grid, with nine text and image pairs in one user message. Each pair started with Tile {i}: Does this image contain any visible part of {target}?. The settings were reasoning.effort=low, detail=high, a strict schema for nine {tile, match} records, max_output_tokens=2048, and store=false. I didn’t set temperature or cache TTL.

The {target} placeholder was bicycle, bus, or fire hydrant. The requests didn’t include labels, expected positive counts, or other models’ answers. Nothing was retried, and I didn’t change the prompts or the threshold after seeing results.

Speed, and the order things ran in

SetupMin, sMedian, sp90, sMax, s
Apple Foundation Models4.8935.0185.4776.896
Decisions + GPT-6 Luna0.7001.2333.8824.341
Apple ∪ Decisions5.6266.3479.1269.387
GPT-6.1 Terra2.3272.7753.0783.482

Decisions had the shortest median, but look at its tail: the p90 sits above Terra’s maximum. Most grids came back fast, and a few took much longer. The p90 here is nearest-rank, the 27th of 30 sorted values, across different grids in one pass.

The timers don’t measure quite the same thing. Apple’s grid time includes starting the native processes, writing the crop files, and nine sequential calls. The API timers cover the HTTP request and parsing, not image encoding.

All runs happened on the same night, one after another: Apple, then Decisions, then Terra, a few minutes apart. Apple refused one cell, and the three Apple cells left over after that refusal ran between Decisions and Terra. The section on the refusal explains why. Every pass went ten bicycle grids, then ten bus, then ten fire hydrant. I didn’t randomize or interleave the setups or control network load. There was no warm-up, so cold starts are in the data.

Where the misses are

In this post, an “error” means disagreement with the frozen COCO labels.

SetupPositives foundMissesFalse positivesUndecided answersCorrect cells
Apple Foundation Models59/903011238/270
Decisions API + GPT-6 Luna83/90710262/270
Apple ∪ Decisions84/90620262/270
GPT-6.1 Terra87/90320265/270

Apple’s problem is recall. It almost never said yes by mistake, with one false positive, but it missed 30 of 90 positives.

The 57 flagged cells are where things get interesting. They all stay in the main score, and all of them are COCO positives, so the only possible errors on them are misses and refusals.

SetupErrors on 57 flagged cellsMissesRefusalsErrors on the other 213 cells
Apple Foundation Models2120111
Decisions API + GPT-6 Luna6602
Apple ∪ Decisions5503
GPT-6.1 Terra2203

Terra’s whole lead over Decisions comes from those hard cells, 2 errors against 6. On the other 213 cells, Decisions made 2 errors and Terra made 3.

Pairing the grids tells the same story. Terra and Decisions both got 21 grids right, only Terra got 5, only Decisions got 2, and neither got 2. An exact two-sided binomial test on the 5 against 2 discordant grids gives p ≈ 0.45. That split doesn’t show that either setup is better.

I also wanted to know whether the disagreements were model mistakes or label problems. After the main run, an AI assistant inspected the 13 Terra and Decisions disagreements, which fall on 12 unique cells. It compared the exact inputs with the originals, the segmentation masks, and the boxes. The review was not blind and is not independent annotation, and it doesn’t change any score. It sorted the 12 cells like this:

  • 4 look like real model errors, 3 by Decisions and 1 by Terra.
  • 4 are ambiguous because of the crop, the source framing, or occlusion.
  • 2 carry conflicting COCO categories for the same object (bus and truck, bus and train).
  • 1 is a bicycle pictogram painted on a road.
  • 1 is a cropped wheel fragment the review couldn’t resolve.

The review found no missing annotation. The labels and scores stay unchanged.

Apple refused one cell

On cell 6 of grid fire-hydrant-03 (COCO image ID 566923), Apple returned May contain unsafe content. I didn’t retry the call or turn the refusal into a “no”. The undecided cell stays in the denominator of 270 and doesn’t count as correct.

My first adapter stopped the grid there, so I ran the three remaining cells later with the same prompt. The grid’s time is the sum of the two segments, without the gap. The cloud APIs, for their part, finished every request without errors.

Adding Apple to Decisions bought one grid and nothing else

The question here is whether the two cover for each other. A cell in the union is positive if Apple or Decisions said yes. A refusal stays undecided: true OR unknown is true, and false OR unknown is unknown. Decisions said yes on the cell Apple refused, so the union has 270 decided answers.

Compared with Decisions alone, the union gained one true positive and one false positive. Correct cells stayed at 262 of 270. That one extra true positive happened to complete a grid, so full grids went from 23 to 24. Meanwhile the estimated median time went from 1.23 s to 6.35 s per grid, and the API cost stayed the same.

I didn’t test a cascade or a fallback to Terra on low Decisions confidence. Such a fallback would have missed at least one case: Decisions gave cell 1 of grid bus-01, a positive bus cell, probability=0.0.

What it cost, and why caching made it worse

The surprise in the cost numbers is that caching made things more expensive. Every Terra request wrote its prompt to the cache, and no request ever read from it. With no cache reads, the writes cost more than no caching would have. The scenario table further down shows the difference.

I checked prices on 2 November 2026, at standard rates. The Decisions API with GPT-6 Luna costs $0.10 per million input tokens, and output is free. GPT-6.1 Terra costs $1 per million input tokens and $5 per million output tokens.

APIInputOutputCache readsCache writes (part of input)Cost for 30 grids
Decisions + GPT-6 Luna69,990000$0.006999
Responses, GPT-6.1 Terra39,3002,873039,210$0.0634675

Terra’s output includes its 273 reasoning tokens. Cache writes are already part of the input count, so I don’t add them again. The Apple adapter doesn’t return token usage. Per image, including the prompt text, Decisions used about 259 input tokens (69,990 / 270) and Terra about 146 (39,300 / 270).

Which cache rate applies is less clear than I’d like. I priced cache writes at the 30m rate, $1.25 per million for Terra, because my requests didn’t set a TTL. The prompt caching guide documents 30m as the default for GPT-5.6 and newer models and lists 24h only for older models. The raw responses, however, returned the legacy field prompt_cache_retention: 24h and no prompt_cache_options.ttl. I haven’t checked that discrepancy against a bill.

At the documented default, Terra’s cost for 30 grids is:

90 regular input tokens × $1 / 1,000,000
+ 39,210 cache-write tokens × $1.25 / 1,000,000
+ 2,873 output tokens × $5 / 1,000,000
= $0.0634675

Two other pricing scenarios use the same usage. Neither is a confirmed charge.

ScenarioTerra, 30 gridsTerra, 1,000 grids
Cache writes at the 30m rate (main table)$0.0634675≈$2.12
Cache writes at the 12h rate$0.112480≈$3.75
No caching, all input at the standard rate$0.053665≈$1.79

The guide says that explicit-only caching mode with no breakpoints creates no cache writes. I didn’t test that setting.

How much memory Apple’s model used

During the first five grids (45 Apple calls), I took 28 samples of MEM from macOS top and summed the footprint of every TGOnDeviceInferenceProviderService and modelmanagerd process. The sum was 459.9 MB before the run, 775.9 MB at the highest sample, and 735.1 MB after. The true peak could be higher. macOS also doesn’t fully count shared weight mappings and caches, so these numbers aren’t the model’s full RAM budget.

Would a bigger model help?

After seeing the main results and the visual review, I wanted one more answer: does a more expensive model in the same API do better on the same grids? So I ran GPT-6.1 Sol. Keep in mind that I chose this comparison after the fact. Sol ran about an hour after Terra with the same request and only the model ID changed, so it also ran at reasoning.effort=low.

GPT-6.1 TerraGPT-6.1 Sol
Full grids right (95% interval)26/30 (70–95%)24/30 (63–91%)
Correct cells265/270262/270
Positives found87/9087/90
False positives25
Median wall time per grid2.78 s3.08 s
Cost per 1,000 grids at the 30m cache-write rate≈$2.12≈$4.26

Sol cost twice as much and made three more false positives. Both models missed 3 positives, and 2 of those misses are the same cells: cell 5 of grid bicycle-09 and cell 3 of grid bus-04. Sol also missed cell 2 of bus-04, which Terra found, and Terra missed cell 1 of bus-08, which Sol found. Cells are numbered 1 to 9, left to right, top to bottom. I didn’t run the visual review or the flagged-cell breakdown on Sol.

Sol’s measured input usage matched Terra’s: 39,300 tokens, 39,210 of them cache writes. At Sol’s prices of $2 per million input tokens, $10 per million output tokens, and $2.50 per million 30m cache writes, 30 grids cost $0.127665. With 12h cache writes at $5 per million, 1,000 grids would cost about $7.52. With no caching, about $3.60. All 30 Sol responses returned the same prompt_cache_retention: 24h field as Terra’s, with no prompt_cache_options, so the same pricing caveat applies.

There was no Terra control run at the same later time. The gap of 26 against 24 grids comes from 2 grids that only Terra got right and 0 that only Sol got right, and it may be run-to-run variance. So at low effort Sol didn’t do better than Terra, and with 30 grids the two can’t be separated.

What I took from it

Terra was the most accurate setup: 26 of 30 full grids and 265 of 270 cells. Decisions got 23 grids and 262 cells at a much lower estimated API cost. Its median time was shorter, 1.23 s against 2.78 s, but it had a long tail, with a p90 of 3.88 s. The full-grid intervals for Terra (70–95%), the union (63–91%), and Decisions (59–88%) overlap widely, so 30 grids don’t separate them. Apple’s interval (7–34%) doesn’t overlap any of them. Apple’s gap is the one result here that is clearly real.

For my small workload I picked Decisions alone. Its accuracy can’t be told apart from Terra’s at this sample size, and it costs about a tenth as much. Adding Apple gave one more full grid, no net correct cells, one more false positive, and about 5 s more per grid, so the data doesn’t show that Apple adds accuracy. Sol doesn’t change that choice either.

I started out wanting to know whether the free, local model was enough. For this job it wasn’t, and the cheapest cloud call turned out to be the sensible default. Paying more bought me very little that 30 grids can show.

Sources and testing context

The test ran on 2 November 2026 on a Mac mini M4 (Mac16,10) with 16 GiB of unified memory and macOS 27.0, build 26A428, the public release Apple announced on 14 September 2026. The preparation used Python 3.12.13, Pillow 13.0.0, NumPy 2.5.4, and pycocotools 2.0.11. Prices were checked the same day. The prompts are above. The harness, the manifest, and the raw responses aren’t published.