Per node settings

I have a node where I need to limit its access to only certain roles. I turned on per node permissions on the content type. Then I went to the node and clicked on the "Access Control tab". I changed which roles should be able to view this piece of content (normally everyone can see this node type, but I need to limit it to just certain roles). Clicked submit. Yet the page is still viewable even when you are not logged in.

So I tried adding a new content type. Didn't give access permissions to anonymous users or authenticated users - just admin. Then I created a piece of content in it. Checked and I can access it when I'm not logged in.

It's very odd, as we have a large amount of content already on the site that is controlled by this module and the access works correctly. I can't figure out why it's not working for these new items.