Directly addressing the reader is jarring.
Depending on content, doing so feels presumptive, accusatory, didactic. It tears down what theater folks call the "fourth wall," the invisible barrier between actors and audience, in a way that serves to fail what we are writing.
An example that may illustrate the point: on more than one occasion, when I have become consciously aware of my physical movements in the process of doing them — i.e., descending a staircase, or tying my sneakers, or fastening a clasp on a necklace — I immediately become physically unable to complete that action. The transformation of the automatic into the thought-FULL staggers me, my weight shifted into thin air, so if the handy banister is not secured fastened to the wall I'd tumble to the bottom with a broken leg or worse.
Readers don't want to be reminded that they are reading. It pulls you out of the story. And this is true in categories outside fiction as well.
An important caveat: sometimes the second person really, really works. The first example that comes to mind is Ta-Nehisi Coates' wonderful Between the World and Me. The book is epistolary in structure, written from father to son. But Coates' strategy draws us even more into his writing, his truth, in an uncomfortable but perfect way. Italo Calvino's classic If on a Winter's Night a Traveler is a metaphysical novel that employs the second person to draw the reader — who is the reader character in the book — even deeper into the experience. Jamaica Kincaid's story "Girl" likewise succeeds principally because it is told in second person.
As always, the questions are: what is the intention of the piece? What is its structure and organization? And: what do these choices do for the reader's experience?