There are multiple steps required to add a new filter. You first need to create your own python package. You then need to install it on the system running OpenStack and add it to enabled_filters and available_filters under the filter_scheduler section of the nova configuration.
Assuming you create a package named nova_filter_example with a filter named RandomFilter, you would need to change your nova.conf to look something like this.
[filter_scheduler]
available_filters = nova.scheduler.filters.all_filters
available_filters = nova_filter_example.RandomFilter
enabled_filters = ComputeFilter,RandomFilter
I created a small example example package that might help. You can find it here. It includes steps on how to deploy and test the custom filter in a devstack environment.