I have caught myself wishing for or making plans that someone else has told me are impossible. This is likely to have happened to you too. For example, when it comes to career choices, some are more especulative than others. Some careers rely heavily in luck, and people often believe these to be impossible dreams. For example, becoming a famous singer or dancer, being a Hollywood star, or being a professional athlete are extremely difficult things to achieve.
As an example, I would say that "my castle" would be becoming a famous writer. If there were no limitations on my life, I would spend all my time writing fiction and poetry, and I would make a living out of it. I would also be a famous person, and everyone would know my name. I would travel for conferences and book presentations, and my ideas would be shared all over the world.
I believe that it is not bad to "build castles in the air." First of all, there is always a possibility that these dreams will come true, since there are some people who have achieved it. Moreover, a "crazy" dream can make you be more disciplined about the things that you want to achieve, which would be good for your life even if the dream is never accomplished. Finally, having a goal can give direction to your life, and if the dream becomes too difficult, you can always adjust your expectations.