Custom board on System Workbench is a quick but not really precise way to create and test a project for your board. CubeMX provided a complete interface to configure your chip then to generate a project with configuration and drivers code (Make sure, you’re using the latest version of CubeMX ! There are known issues in the generated project for SW).
About your issue :
The OpenOCD script is a configuration file (*.cfg) use by OpenOCD (On-chip debbuger) to identify your debug device and chip. With custom board, a *.cfg file should be generated into your project directory. Is this file existing in your project directory ?
1/ It it exists, go to Run > Debug configuration..., then select your debug configuration and go to the “Debugger” tab. In “Script” section, select “use local script” and browse the *.cfg file.
2/ If not, make sure you’re using the latest version of the SW, then try to create a new project again with your newly created board selected (It should be listed in STM32F4 series). The OpenOCD script should be generated this time. You can copy the *.cfg file into your first project then do as how I describe in 1/.
As I cannot reproduce your issue, I don’t if what I’m suggesting will solve your problem...
Hope it helps you.
PS : Find my OpenOCD script for STM32F4DISCOVERY in attachement if needed.