Yes, if you do not wish to allow your constituents to add service hours on the Donor Portal you can disable it. From the Donor Portal Info page, change the toggle for Disable Ability to Add Time in the Portal to YES. If you have the Volunteer Hours item checked, your constituents will still be able to see their volunteer time, just not add any new time.

If you want to completely hide information about volunteer time, uncheck Volunteer Hours.