I agree, I wouldn't give back the money. As the old saying goes Time is Money, how much is your time worth? Personally if I was in you situation before I even agreed to start creating a website/logo I would ask for a non refundable deposit of half the agreed upon price just to make sure that if they decided to back out you atleast get some compensation for time put into the project.
Think of it this way, a breeder always asks for a deposit for an animal if you want to reserve it. If you have to back out for whatever reason, that deposit is gone. The breeder keeps it for the time he spent keeping, houseing, feeding the animal while you were getting the rest of the money, it's only fair.
1.2.1 Ball Pythons
1.0 Cali King Snake (Weebl)
0.0.1 Black Breasted Leaf Turtle (Hootie)