When it's a named street. For example, if you were wrtiting down or texting someone directions without a specific street name, you'd write "Turn left on that street." But if you were giving a SPECIFIC street name, you'd say, "Turn left on Oakwood Street."
Another example, The word ‘hill’ is not a proper noun, but Gellert Hill is because it’s the name of one specific hill.
In case you haven't got it by now, only capitalize if you're being specific about the name.