A/B test on manual versus AI generated ad creative for seller acquisition

Set the marketing question aside for a second. Ads went out showing a house that does not exist, to sellers, and two of them referenced it on a call. Depending on how the image was presented and which state the ads ran in, misleading advertising rules and in some states real estate advertising rules can reach that, and if the ads run under a license the exposure is different again. That deserves a real conversation with an attorney in your state before Group C runs rather than after. Separate from that, the Group B copy was written by a model optimizing for clicks. Of course it drew people with a different understanding of the offer. That is the mechanism, and it is independent of the image question.

The attribution problem you named is bigger than the creative question. With a 5 to 7 week lead to contract cycle and a 90 day test, roughly the last third of your spend cannot have produced a contract yet. Group B may have contracts in flight you have not counted. Drawing the honesty conclusion from 1 versus 0 is premature when the window cannot hold the cycle.

The two callers mentioning a house that doesn't exist would have ended it for me, no question. Not as a compliance thing, as a first impression thing. You get one.

You are comparing 6 creatives against 34 and calling it manual versus AI. It is also low variant count versus high variant count, and the platform's optimizer behaves differently when it has 34 things to test. Group B may be winning on cost per lead purely because the algorithm had more to work with, and you would get the same effect from 34 manual creatives if you could produce them. That confound has to come out before Group C means anything. Run the next test with equal variant counts.

@luis_carrera is right and it's the strongest point in the thread.

On the actual decision, I'd bid on cost per conversation. I know you don't want to rebuild tracking. Every time I've optimized on the cheap upstream metric I've ended up paying for volume that cost me labor downstream, which is the same shape as scope creep. You spend the savings answering calls.

The rebuild is a week of annoying work and then you have the number that matters for years.

How are you defining conversation to price? If it's your own judgment on a call, that's the metric you're about to bid a budget against and it isn't defined anywhere but your head. Write the definition down first, then rebuild tracking.

The two agencies pitching AI creative at a lower retainer are pricing on production cost, which is exactly the thing that got cheap. What did not get cheap is knowing which offer to make and reading what the leads mean. If their pitch is mostly about volume of variants, you would be paying a retainer for something you just did yourself in four hours. On Group C, run it. Real photos with AI assisted copy and layout is the configuration most shops will land on anyway, and it is better to find out on $2,100 of your own money. Match the variant count per @luis_carrera and keep the price conversation definition @nataliehsu asked for written down before you start.