Book Table of Contents showing unpublished pages

Problem/Motivation

I'm using the Book Table of Contents block and I have some unpublished pages that I'd like to keep in the book, but not show in the side menu for users that don't have permission to view unpublished content.
Is there a way to stop those pages from showing up for those roles that don't have the permission?

Steps to reproduce

Create book content, unpublish 1 or more pages, note how they show up in the Book Table of Contents block.