Traditional simulations of complex mechanical deformation are technologically crucial and computationally expensive. Developing comparably accurate models with lower computational cost can enable more robust design and uncertainty quantification, as well as exhaustive structure-property exploration. Currently, high-throughput experimental techniques and microscale simulators can produce quantities of data that overwhelm traditional constitutive modeling methods and can provide sufficient data to train neural networks. As a modeling technique, neural networks are flexible in that their graph-like structure can be rearranged and functions of their nodes can be adapted to suit particular applications, such as image processing and time integration. On the other hand, Gaussian process models can outperform neural networks in the data limited regime. In this talk, I will discuss: how we construct input spaces and architectures to capture how microstructure affects mechanical outcomes given observations of the initial microstructure; how we draw upon classical constitutive theory to make predictions of the individual stress response that satisfy fundamental physical constraints, and, how we represent the time evolution of inelastic materials.