Automated analysis of strong gravitational lensing images has become an important problem with a huge amount of data expected to come in from future ground- and space-based astronomical surveys, like LSST and WFIRST. Current popular methods include maximum likelihood estimation and (ordinary) artificial neural networks. While these methods provide a way to estimate lensing parameters efficiently, they only provide point estimates of outputs with no measure of uncertainty. They are also prone to overfitting and hence require careful training and regularization measures. Bayesian Neural Networks hold the potential to address the above two drawbacks. Bayesian Neural Networks train a distribution over weights, which can be used to provide a measure of uncertainty in the model as well as noise in the data. The output for a given input is an average across all trained weights, and so is less prone to overfitting. Here, we present our ongoing work to apply Bayesian Neural Networks to the problem of parameter estimation for strong lenses. We present results comparing our predictions to that of ordinary neural networks. We also mention our next steps to improve the model outputs and analyze the aleatoric and epistemic uncertainties.