Traditional simulations of complex mechanical deformation are technologically crucial and computationally expensive. Developing comparably accurate models directly from data with lower computational cost can enable more robust design, uncertainty quantification, and exhaustive structure-property exploration. We have been developing neural network models that are guided by traditional constitutive theory, such as tensor function representation theorems to embedded symmetries, and also exploit deep learning to infer intrinsic microstructural features. Neural networks are flexible in that their graph-like structure can be rearranged and functions of their nodes can be adapted to suit particular applications, such as image processing and time integration. This talk will describe the architecture and demonstrate the efficacy of neural networks designed to model the response of elastic-plastic and viscoelastic materials with pores. inclusions or grains based solely on observable data.