The upcoming O4 observing run of the LIGO, Virgo, and KAGRA gravitational-wave (GW) interferometers, scheduled to start in 2022, has the potential to detect up to tens of binary neutron star mergers. The associated kilonovae from these mergers, when observed in the electromagnetic spectrum, can reveal information about r-process nucleosynthesis, the neutron star equation of state, and can be used as standard sirens to probe the Hubble constant. Most prompt GW follow-up efforts to date have been in optical wavelengths; however, kilonovae emission in the optical is predicted to be short-lived and strongly beamed. In the infrared, emission is expected to be less dependent on geometry and viewing angle, and to remain brighter for longer. The Wide-Field Infrared Transient Explorer (WINTER) is a near-infrared instrument installed on a dedicated 1-m telescope currently being commissioned at Palomar Observatory and will conduct a time-domain survey aimed at discovering and observing kilonovae in the infrared, providing a unique opportunity for multimessenger observations. In this talk, we examine the advantages of follow-up observations in the infrared and present prospects for kilonova discovery using WINTER during O4.