The word "unnerved" would suggest that the narrator's mind is disturbed, particularly since Usher does not seem to have heard the metallic sound. However, at the end, the reader finds out that Usher has known all along that Madeline was not really dead, that she would open the metal doors of the crypt (hence the hollow, clangorous, metallic sound) and walk again into his room.