how to set the Material theme's custom properties?

Check this documentation page: https://vaadin.com/docs/v14/flow/theme/theming-crash-course.html, especially the part Using Custom CSS Properties, that should have some related info. Since Notifications are Overlays, you might also want to look into this: https://vaadin.com/docs/v14/flow/theme/tutorial-theming-overlay.html. Long story short, typically you want to set custom properties on the html element in your .css file. Whether you want to override --material-background-color or just the color used by Notifications depends if you want to affect all other usages of --material-background-color or not - if you do, just overriding the custom property on html level should be the way to go. Otherwise, you’ll need to target Notifications only, and there you’ll need a theme module for the [part~="overlay"] as documented on the Theming Overlay Components page.