The 3.5 percent is figured on the full purchase price of the building, all four units. On a $500,000 fourplex that's $17,500 down, and you'd be borrowing roughly $482,500 plus the FHA mortgage insurance premium that gets added into the loan. Nothing in the loan splits the property into your part and the tenants' part. The rents matter on the income side, they help you qualify for the payment, and they don't shrink the down payment.
Two things travel with that number. FHA lending is for owner-occupants, so you sign a commitment to live in one unit, normally for at least a year. And on three and four unit properties FHA applies a self-sufficiency test: the appraiser estimates market rent for every unit including the one you'll live in, and 75 percent of that total has to cover the whole monthly payment, meaning principal, interest, taxes, insurance and mortgage insurance. Lots of fourplexes fail that test in pricier markets, which is why plenty of first-time buyers land on a duplex instead.
The line that surprises people is reserves. Lenders often want a few months of the full payment still sitting in your account after closing on a three or four unit purchase, so your cash need is more than down payment plus closing costs. Before you get attached to a building, ask your lender to put the reserve requirement, the mortgage insurance amount and the occupancy language in writing. Program terms shift and vary by lender.