We present a notion of bisimulation that induces a reduced network which is semantically equivalent to the given neural network. We provide a minimization algorithm to construct the smallest bisimu- lation equivalent network. Reductions that construct bisimulation equiv- alent neural networks are limited in the scale of reduction. We present an approximate notion of bisimulation that provides semantic closeness, rather than, semantic equivalence, and quantify semantic deviation be- tween the neural networks that are approximately bisimilar. The latter provides a trade-off between the amount of reduction and deviations in the semantics.