According to the context, the word "decaying" in this sentence suggests a lack of life.
The description in this text is the characters' memory of scenes they used to imagine taking place there in the past. They are looking at their surroundings and picturing the life and activities which could have happened in this bay, based on local tales they were told:
- "It was a scene we had acted out as youngsters"
- "The oldest stories told..."
- "building an imaginary bridge"
This is all in the characters' imagination however, and in reality, this fishing camp is abandoned and decrepit. You can tell from the other hints in the text which are in the same lexical field as decay:
- "rotting"
- "old"
- "weathered"
To decay means to decompose, to lose health or vitality. The author is depicting a contrast between the characters' vivid imagination and the place's real lack of life.