We present the Photon Simulator (PhoSim), a simulation tool for generating astronomical images. PhoSim uses an ab initio physics-based approach with a multi-threaded Monte Carlo method. PhoSim follows the path of individual photons from astronomical sources, through the atmosphere, mirrors, lenses, and sensors, and then generates images by collecting electrons in pixels. It has a number of space-based and ground-based astronomical observatories implemented. The simulator uses appropriate hydrodynamics, elasticity theory, and electrostatic physics to describe the interactions at covers the IR, optical, UV, and X-ray wavebands. PhoSim can also run on large-scale computing platforms to simulate large surveys. PhoSim can be used to design new telescopes, simulate future observations, and perform detailed astronomical measurement systematic studies for either advanced image processing and machine learning algorithms. PhoSim is publicly available, and is periodically updated.