Why wouldn't you want a Supreme Court Justice to be empathetic? We already have our quota of psychopaths on the court.
Most of the justices have lived lives that insulated them from adversity, and this is bound to color their worldview and the way they interpret the law. Frankly, I don't think the Supreme Court really understands or much cares about how its decisions impact ordinary people. Other than their servants, they rarely encounter anyone ordinary.
Someone like Roberts can claim that he's just calling balls and strikes because he reckons that his worldview is the only one. He doesn't get it that his background and life experience color his interpretation of the law as much as anyone else's.