Hyperopt list of values per hyperparameter

Question

I'm trying to use Hyperopt on a regression model such that one of its hyperparameters is defined per variable and needs to be passed as a list. For example, if I have a regression with 3 independent variables (excluding constant), I would pass hyperparameter = [x, y, z] (where x, y, z are floats).

The values of this hyperparameter have the same bounds regardless of which variable they are applied to. If this hyperparameter was applied to all variables, I could simply use hp.uniform('hyperparameter', a, b). What I want the search space to be instead is a cartesian product of hp.uniform('hyperparameter', a, b) of length n, where n is the number of variables in a regression (so, basically, itertools.product(hp.uniform('hyperparameter', a, b), repeat = n))

I'd like to know whether this is possible within Hyperopt. If not, any suggestions for an optimizer where this is possible are welcome.

So you want to make a grid search or what do you mean with the cartesian product? — Sebastian Baum, Aug 08 '20 at 16:21
Gridsearch is just one of many optimization methods with respect to search space, so I'm not sure why you're asking about it specifically. You make it sound like the choice of search space is dependent on the optimization method. — Always Right Never Left, Aug 08 '20 at 16:40
I am sorry but i am a little confused, what you want to achieve. You want a dynamical hyperopt search space, where you just define a search space for one variable and it will automatically implement it for all variables for your regression? — Sebastian Baum, Aug 08 '20 at 16:49
Right, so here's an example: say we have a binary (for simplicity) hyperparameter that can be either 0 or 1. If it was applied to all variables, the search space would be [[0], [1]]. What I want instead is for it to applied individually and independently per variable. Say we have 2 variables, then the search space would be [[0, 0], [0, 1], [1, 0], [1, 1]]. This is straightforward to implement when using gridsearch (and you don't need Hyperopt for it), but it is expensive to evaluate all the points, so I want to use something more efficient like Bayesian optimization (in this case hyperopt.tpe) — Always Right Never Left, Aug 08 '20 at 17:20
In other words, it would be like setting multiple hyperparameters as `hp.uniform('hyperparameter', a, b)` and then recombining them into a list to feed to the regression as a single hyperparameter. I tried that and the kernel just died after some time, I don't think this is due to search space blowing up but rather that such manipulations can't be handled natively by Hyperopt. — Always Right Never Left, Aug 08 '20 at 17:35
I don't understand your need. Why not just do `hp.uniform('hyperparameter', a, b)` three times if you have three variables? hyperopt will then apply its bayesian optimizer on their combinations. — kg_sYy, Aug 15 '20 at 14:07

score 0 · Answer 1 · answered Aug 15 '20 at 14:34

As noted in my comment, I am not 100% sure what you are looking for, but here is an example of using hyperopt to optimize 3 variables combination:

import random

# define an objective function
def objective(args):
    v1 = args['v1']
    v2 = args['v2']
    v3 = args['v3']
    result = random.uniform(v2,v3)/v1
    return result

# define a search space
from hyperopt import hp

space = {
    'v1': hp.uniform('v1', 0.5,1.5),
    'v2': hp.uniform('v2', 0.5,1.5),
    'v3': hp.uniform('v3', 0.5,1.5),
}

# minimize the objective over the space
from hyperopt import fmin, tpe, space_eval
best = fmin(objective, space, algo=tpe.suggest, max_evals=100)

print(best)

they all have the same search space in this case (as I understand this was your problem definition). Hyperopt aims to minimize the objective function, so running this will end up with v2 and v3 near the minimum value, and v1 near the maximum value. Since this most generally minimizes the result of the objective function.

score 0 · Answer 2 · answered Sep 11 '20 at 23:46

You could use this function to create the space:

def get_spaces(a, b, num_spaces=9):
    return_set = {}
    for set_num in range(9):
        name = str(set_num)
        return_set = {
                **return_set,
                **{name: hp.uniform(name, a, b)}
                }
                
    return return_set

Asclepius · Answer 3 · 2022-02-02T19:43:32.660

I would first define my pre-combinatorial space as a dict. The keys are names. The values are a tuple.

from hyperopt import hp
space = {'foo': (hp.choice, (False, True)), 'bar': (hp.quniform, 1, 10, 1)}

Next, produce the required combinatorial variants using loops or itertools. Each name is kept unique using a suffix or prefix.

types = (1, 2)
space = {f'{name}_{type_}': args for type_ in types for name, args in space.items()}

>>> space 
{'foo_1': (<function hyperopt.pyll_utils.hp_choice(label, options)>,
  (False, True)),
 'bar_1': (<function hyperopt.pyll_utils.hp_quniform(label, *args, **kwargs)>,
  1, 10, 1),
 'foo_2': (<function hyperopt.pyll_utils.hp_choice(label, options)>,
  (False, True)),
 'bar_2': (<function hyperopt.pyll_utils.hp_quniform(label, *args, **kwargs)>,
  1, 10, 1)}

Finally, initialize and create the actual hyperopt space:

space = {name: fn(name, *args) for name, (fn, *args) in space.items()}
values = tuple(space.values())

>>> space
{'foo_1': <hyperopt.pyll.base.Apply at 0x7f291f45d4b0>,
 'bar_1': <hyperopt.pyll.base.Apply at 0x7f291f45d150>,
 'foo_2': <hyperopt.pyll.base.Apply at 0x7f291f45d420>,
 'bar_2': <hyperopt.pyll.base.Apply at 0x7f291f45d660>}

This was done with hyperopt 0.2.7. As a disclaimer, I strongly advise against using hyperopt because in my experience it has significantly poor performance relative to other optimizers.

score -1 · Answer 4 · answered Aug 08 '20 at 17:45

Hi so I implemented this solution with optuna. The advantage of optuna is that it will create a hyperspace for all individual values, but optimizes this values in a more intelligent way and just uses one hyperparameter optimization. For example I optimized a neural network with the Batch-SIze, Learning-rate and Dropout-Rate:

The search space is much larger than the actual values being used. This safes a lot of time instead of an grid search.

The Pseudo-Code of the implementation is:

def function(trial): #trials is the parameter of optuna, which selects the next hyperparameter
    distribution = [0 , 1]
    a = trials.uniform("a": distribution) #this is a uniform distribution
    b = trials.uniform("a": distribution)

    return (a*b)-b
    #This above is the function which optuna tries to optimze/minimze

For more detailed source-Code visit Optuna. It saved a lot of time for me and it was a really good result.

Hyperopt list of values per hyperparameter

4 Answers4