Some afternoons The Giver sent him away without training. Jonas knew, on days when he arrived to find The Giver hunched over, rocking his body slightly back and forth, his face pale, that he would be sent away.
"Go," The Giver would tell him tensely. "I'm in pain today. Come back tomorrow." (13.79-80)
The Giver is in pain because of his awareness. Through the memories he holds, he knows there is suffering in the world—even without the mystical quality of the memories (that is, the way that they physically affect whoever holds them). It makes sense that The Giver would suffer for his knowledge.
Quote 8
The Giver shook his head and sighed. "No. And I didn't give her physical pain. But I gave her loneliness. And I gave her loss. I transferred a memory of a child taken from its parents. That was the first one. She appeared stunned at its end." (18.34)
The Giver, because he loved Rosemary, didn't want her to feel physical pain. But his decision to give her emotional pain instead may have been even more destructive.
The man smiled. He touched the sagging flesh on his own face with amusement. "I am not, actually, as old as I look," he told Jonas. "This job has aged me. I know I look as if I should be scheduled for release very soon. But actually I have a good deal of time left." (10.38)
It looks like the pleasure-pain connection works here, too; The Giver is wiser for his memories, but has been physically weakened by them as well.