“When I was growing up”
Whether a piece of writing is formal or informal depends on the choices of vocabulary, the way it is organized and the tone.
Informal languages tend to be more personal than formal language by using first-person pronouns such as ‘I’ or ‘We’, is more casual and spontaneous, and may use colloquialisms, contractions, and the active voice instead of the passive one. Likewise, the words in this style tend to be short, or words that are common while texting, in friends meetings, etc.
“when I was growing up” uses informal language: it is personal ("I"), casual, and the words can be used in informal settings and be understood by anyone. If you wanted to say the same phrase in a more formal way, "At the time of my childhood" would be a good option.