This presentation examines the use of Physics-Informed Neural Networks (PINNs), Variational Physics-Informed Neural Networks (VPINNs), Deep Ritz methods, and First Order System Least Squares (FOSLS) combined with stochastic quadrature rules, to solve parametric partial differential equations (PDEs). It begins by introducing parametric PDEs and how these neural network techniques can be used to solve them. The presentation then delves into the challenges of solving these PDEs, including optimization, regularity, and integration. It points out that while PINNs using strong formulations may have trouble with singular solutions, they handle integration better than weak formulation methods like VPINNs or Deep Ritz. To address these integration challenges, we propose the use of unbiased high-order stochastic quadrature rules for better integration and Regularity Conforming Neural Networks to deal with complex solutions and singularities. Finally, the presentation discusses the broader significance of this research for solving parametric PDE problems and suggests directions for future research, and how FOSLS and PINNs may work better than VPINNs and Deep Ritz in different cases.