Assuming the question is "Cuándo vamos a la biblioteca", this is asking "When are we going to the the library?" So, the response would likely be something like one of these:
Vamos a biblioteca a las cuatro.
Vamos a biblioteca el miercoles.
Or, more simply, something like these:
Vamos en dos horas.
Vamos mañana.
Vamos el viernes.