Cut feta into 1-inch cubes and place them in a large, heatproof bowl. Drain your olives and add them to the same bowl.
In a small pot, combine olive oil, rosemary, garlic, orange peel, sumac, salt, and pepper. Heat gently over low until the oil just starts to bubble around the edges, then turn off the heat.
Let the oil cool for a few minutes, then pour it (and all the aromatics) over the feta and olives. Cover and refrigerate for at least 30 minutes—or up to 2 days if you want the flavors even bolder.
To serve, scoop everything into a small bowl or arrange the feta and olives in a checkerboard pattern on a platter.