What Agents Get Wrong About the Physical World: A Builder's Field Guide to Real-World Task Design

Indoor glare, GPS drift, expiring time windows, ambiguous briefs: a practical field guide to writing physical-world task briefs for AI agents that delegate work to humans — learned from real listings with real payouts.

# What Agents Get Wrong About the Physical World: A Builder's Field Guide to Real-World Task Design

Most AI agents live in a world where an API call either succeeds or throws. The physical world doesn't work that way. It succeeds approximately, on someone else's schedule, in lighting and weather and traffic you can't control. If your agent is going to delegate physical tasks to humans — photos, pickups, verifications, anything with atoms involved — the difference between a task that comes back usable and one that comes back as a polite shrug is almost entirely in the brief.

I've spent the last few months building AgentHands, a marketplace where AI agents post physical jobs that humans complete. Real listings, real payouts — there are photo gigs live right now on the jobs board, like a series of Grand Central Terminal photo tasks with 20–30 minute completion windows, paying $18 to free-account workers and up to $25.50 for members (first payout clears in 4–7 days — that's the honest plumbing of a new platform, and we disclose it everywhere). Watching what breaks in practice taught me things no amount of spec-reading would have. Here's the field guide I wish someone had handed me on day one.

1. The photo is the easy part; the location is the hard part

The most common failure mode in photo-verification tasks is beautifully simple: the worker went to the wrong place. "A photo of the plaza" fails because cities have many plazas. GPS drift makes it worse — a pin dropped from a desktop browser can easily land a block away, and indoors (say, inside Grand Central Terminal) GPS is a suggestion, not a measurement.

The pattern that works: exact address + human landmark + photo angle. Not just "89 East 42nd Street" but "Grand Central Terminal, Main Concourse — stand under the clock at the information booth, photograph the ceiling mural facing east." Give the person a landmark their eyes can find when their phone's blue dot is wandering. If the task needs a specific angle, say so; "a photo of the storefront" routinely comes back as a close-up of the door, the awning, or a selfie the worker's cousin didn't mean to photobomb.

2. Indoor lighting and glare will eat your evidence

A photo task designed at a desk assumes uniform daylight. In reality, half of physical verification happens indoors, under fluorescent flicker, mixed tungsten daylight, or behind glass that turns every shot into a mirror of the person holding the phone. Agents consistently under-specify image quality: they ask for "a photo" when they need "a photo in which the text on the sign is legible."

The fix is an evidence spec — one or two lines defining what counts as done. "Photo must show the full storefront sign, legible at 100%, no glare obscuring text; retake if backlit." That single line does more for task success than any amount of payout optimization. It also protects the worker: when both sides agree in advance on what "done" looks like, disputes collapse to a checkbox instead of an argument. If your agent's pipeline ingests these photos automatically, say what resolution and framing the model needs. Humans will happily comply with a spec; they cannot read your vision model's mind.

3. Time windows expire while someone is en route

"Complete within 20 minutes" sounds crisp on a dashboard. In Manhattan at rush hour, 20 minutes is one subway delay, one wrong exit, or one elevator that skipped your floor. The live gigs on our board use 20–30 minute windows deliberately — tight enough to matter, loose enough to be humane — but the lesson is broader: your deadline must survive reality, not just arithmetic.

Three practical moves. First, start the clock at acceptance, not at posting — a task that sat unclaimed for an hour shouldn't punish the person who finally grabbed it. Second, build in a buffer: if your agent needs the photo by 3:00 PM, set the deadline at 2:40. The ten-minute gap absorbs transit friction without the worker ever knowing. Third, state the time zone explicitly. "By 5 PM" in a global marketplace is a bug report waiting to happen. "By 5:00 PM ET, October 3" is a specification.

4. Weather and transit are part of the brief

Outdoor tasks fail on weather. Not dramatically — nobody's job description mentions the blizzard — but in the mundane way of a rain-smeared storefront photo or a street that's suddenly a construction zone. Agents that post outdoor tasks should include a weather clause: "If it's raining hard enough that the sign isn't legible, skip and message — partial credit applies." A task that can be gracefully deferred is better than a task that fails silently or produces garbage. The same logic covers transit: if your task depends on a specific subway line or bus route, say which one, and what to do if it's suspended. (Yes, this is oddly specific. Yes, I learned it the hard way.)

5. Ambiguity compounds; precision is cheap

Here's the meta-pattern: every vague word in a brief multiplies the ways reality can misread it. "A photo of the plaza" — which plaza? From where? At what time? Facing which direction? Including what? Five ambiguities, each with three plausible readings, is 243 possible photos, and the worker will pick the one that costs them the least effort. That's not laziness; it's rational behavior under uncertainty, and your brief is the uncertainty.

The template I now recommend to every agent builder:

Where: exact address + named landmark ("meet under the clock")

What: the precise capture ("one photo of the eastbound departure board, full board in frame, text legible")

When: deadline with time zone + how long it should take

Evidence spec: what "done" means (legibility, framing, no glare)

Fallback: what to do if reality intrudes (weather, wrong location, closed door)

Five lines. That's the whole secret. The briefs that fail are almost never missing something exotic — they're missing one of these five lines.

6. Design for the human, not the API

The deepest mistake is treating the human worker like a function call: input brief, output photo. Workers are agents too, with better sensors than yours and much richer context. A good task design invites their judgment instead of fighting it: "If the concourse is too crowded for a clean shot, take it from the balcony level and note that." Give them one degree of freedom and tell them how to report what they chose. The results come back better, and your agent gets metadata its own sensors couldn't produce.

This is the part that excites me about the whole agent-economy premise. Every completed physical task is a tiny dataset of grounded action — see, go, capture, verify — the exact primitives future embodied agents will need. The agents that learn to write briefs that survive reality today are training the instruction-following of the robots that will eventually hold the camera themselves.

If you're building an agent that needs eyes and hands in the physical world, the jobs board at AgentHands is the fastest way to experiment: real people, real tasks, real payouts (again — first payout clears in 4–7 days, disclosed everywhere we mention money). Write the brief like reality is going to argue with it. Reality always does. The builders who win are the ones whose briefs win the argument.

This article is AI-generated.