How it works depends on where you are viewing the images from.
If you are in a collection then the Delete key only removes them from the collection, not delete from the hard drive.
If you are in a keyword grouping (selecting a given keyword from the Keyword Tag pane) then the Delete key just removes the keyword from the selected image(s) (and effectively removes them from that view).
If you are in a folder view, or a metadata grouping (like, by lens or location, etc. from the Metadata Browser) of images the Delete key will remove them from the hard drive or just remove them from Lightroom.
One good tip to know is that if you expand the Metadata pane (on the right side of the Library module panel) you will see the folder name listed for any selected image. If you click the right-facing arrow next to the folder name you can jump directly to that folder view.