tag:blogger.com,1999:blog-6061887630060661987.post4076341170268576483..comments2019-02-16T18:38:45.386-05:00Comments on dlib C++ Library: A Global Optimization Algorithm Worth UsingDavis Kingnoreply@blogger.comBlogger69125tag:blogger.com,1999:blog-6061887630060661987.post-29769628958345185982019-02-16T08:25:16.596-05:002019-02-16T08:25:16.596-05:00The first few points are just picked randomly.The first few points are just picked randomly.Davis Kinghttps://www.blogger.com/profile/16577392965630448489noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-82442792089050947262019-02-16T04:57:43.798-05:002019-02-16T04:57:43.798-05:00In the animated (video) version of the algorithm, ...In the animated (video) version of the algorithm, how do you obtain the location of the 4th point?Johanneshttps://www.blogger.com/profile/17203427465672281313noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-13777226274139040132019-02-11T18:42:06.406-05:002019-02-11T18:42:06.406-05:00Awesome :)Awesome :)Davis Kinghttps://www.blogger.com/profile/16577392965630448489noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-84621449440271374722019-02-11T04:50:29.408-05:002019-02-11T04:50:29.408-05:00I got the job, by the way \o/
For reference and u...I got the job, by the way \o/<br /><br />For reference and until I do more investigation and a write-up, this optimizer seems to work very well with XGBoost optimizing over five hyperparameters. It can usually find an excellent candidate for a global optimum in considerably less than 50 function calls.Matthttps://www.blogger.com/profile/04236324797918643122noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-36434917508052624472019-01-29T20:59:29.596-05:002019-01-29T20:59:29.596-05:00Sweet :)Sweet :)Davis Kinghttps://www.blogger.com/profile/16577392965630448489noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-49493423611790110742019-01-29T09:29:50.935-05:002019-01-29T09:29:50.935-05:00Thanks Davis, the MP4 version is working well 👍Thanks Davis, the MP4 version is working well 👍Matthttps://www.blogger.com/profile/04236324797918643122noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-18240120073453341532019-01-29T07:37:22.029-05:002019-01-29T07:37:22.029-05:00As for zwep's question, use a closure or lambd...As for zwep's question, use a closure or lambda function.Davis Kinghttps://www.blogger.com/profile/16577392965630448489noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-64785975254027272832019-01-29T07:34:37.681-05:002019-01-29T07:34:37.681-05:00The video is still there when I look at it. It...The video is still there when I look at it. It's hosted on dlib.net. There are even two versions, both of which seem fine: http://dlib.net/find_max_global_example.mp4 and http://dlib.net/find_max_global_example.webmDavis Kinghttps://www.blogger.com/profile/16577392965630448489noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-25812306220395150982019-01-29T06:57:54.822-05:002019-01-29T06:57:54.822-05:00I was going to steal the wonderful video for an in...I was going to steal the wonderful video for an interview presentation, but it now appears to be corrupted... downloading didn't work either :(Matthttps://www.blogger.com/profile/04236324797918643122noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-80468402323955667312018-12-11T15:10:13.876-05:002018-12-11T15:10:13.876-05:00For now I have set all the additional parameters (...For now I have set all the additional parameters (put in a dict) as a global variable. It definitely solve my problem.. but it doesnt 'feel' that great.<br />Could you think of a better solution? (where "better" here is quite arbitrary)zwephttps://www.blogger.com/profile/09240148981899264877noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-91596101994364988152018-12-11T10:02:51.887-05:002018-12-11T10:02:51.887-05:00Hello, I have some trouble using the dlib.find_min...Hello, I have some trouble using the dlib.find_min_global in my NN model...<br /><br />I have setup a class that creates a model object and attaches it to 'self'. Then, a method is called where this model object is trained using self.model_obj.fit_generator from Keras.<br /><br />I have made it in such a way that this method needs additional parameters like learning rate, and others, which I want to approximate using this dlib function.<br /><br />However, there is a 'self' argument in the method by the nature of my class... and I cant seem to find a way to have dlib.find_min_global ignore this first argument in its optimizaiton. Do you have an idea maybe?zwephttps://www.blogger.com/profile/09240148981899264877noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-30371752653510522532018-11-29T07:20:16.150-05:002018-11-29T07:20:16.150-05:00I don't have any numbers to share, but I'v...I don't have any numbers to share, but I've used it plenty on problems of 10 dimension and it works fine. There isn't anything special about 5 vs 10 dimensions.Davis Kinghttps://www.blogger.com/profile/16577392965630448489noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-87995254841296134992018-11-29T04:37:45.296-05:002018-11-29T04:37:45.296-05:00Hi Davis,
For a global optimization algorithm, th...Hi Davis,<br /><br />For a global optimization algorithm, this seems extremely easy to use, and seems to converge fast (from your tests upto 5 dimensions). I have a CFD optimization problem with 10 geometric design variables, and am considering applying this method for finding the global maximum in efficiency. However, since the CFD's are quite computationally expensive, I am trying to be sure it would work well in higher dimensions. Do you have some test results in the range of 6-10 dimensions that you can share with us?<br /><br />Thanks :)praxhttps://www.blogger.com/profile/05305551077908245740noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-68776619694250445492018-08-11T07:30:53.675-04:002018-08-11T07:30:53.675-04:00Sorry for the late reply, for some reason I haven&...Sorry for the late reply, for some reason I haven't been getting notifications of new commends on the blog. <br /><br />Yes, you can do all those more complex use cases by using the global_function_search class directly rather than using find_min_global(). See the documentation for global_function_search.<br /><br />No, there is no built in tool to visualize the surrogate surfaces, if you want to do that it's on you. I had hacked together something to make the video from this blog post, but the code is not readily reusable, so I'm not sharing it. It would be easier for someone to write it from scratch than figure out how to use that bit of hacky code.<br /><br /><br />The extensions I made to the algorithm make it work with things that are not lipschitz. There is a whole discussion in the blog post about functions that are discontinuous. Discontinuous functions are not Lipschitz.Davis Kinghttps://www.blogger.com/profile/16577392965630448489noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-36521416562064596022018-07-13T23:13:30.728-04:002018-07-13T23:13:30.728-04:00Hi Davis! Thank you very much for the sharing.
I ...Hi Davis! Thank you very much for the sharing.<br /><br />I am trying to use your algorithm to tune a Machine Learning program that takes a day to get one data point (each with a specific set of hyper-parameters) . I've already a few data point and would like to know if there is an option in your function to incorporate these data points to predefine the upper bound U(x) (rather than starting from a random point)?<br /><br />Also, is there an option to print out/save the value of the parameter x_i and result y obtained in each iteration so that I can decide whether I should iterate more. <br /><br />My calculation just takes so long to run and those features will help a lot!<br /><br />Thank you.<br />Unknownhttps://www.blogger.com/profile/03452827871556753687noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-13195488982386342612018-07-03T04:44:34.181-04:002018-07-03T04:44:34.181-04:00Is there an easy way to visualise the optimisation...Is there an easy way to visualise the optimisation/surrogate surface? Something similar to what one gets out of skopt plot_objective, plot_evaluations functions?Fahiz Baba Yarahttps://www.blogger.com/profile/02509475568147194357noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-5784628056551123862018-05-30T11:51:46.905-04:002018-05-30T11:51:46.905-04:00Hi Davis!
Thank you for your amazing algorithm!
...Hi Davis! <br />Thank you for your amazing algorithm!<br /><br />I have question for your suggestion using it for hyperparameter tuning:<br /><br />"Shouldn't we know first that the neural network we are using is <br /> a Lipschitz function with input its hyperparameters?"<br /><br />There is already work done proving that neural nets are Lipschitz functions,<br />but not with input their hyperparameters. At least I cannot find any work<br />that proves so...<br /><br />Thanks!Ioannis Athanasiadishttps://www.blogger.com/profile/01749941832682059764noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-86131523205001996492018-05-28T22:25:18.047-04:002018-05-28T22:25:18.047-04:00How can I pass dynamic array to bound1 and bound2?...How can I pass dynamic array to bound1 and bound2? All my approaches are failing...Unknownhttps://www.blogger.com/profile/12397084472218321850noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-80869323547289921572018-05-17T09:30:17.981-04:002018-05-17T09:30:17.981-04:00Yes, it's all derivative free.
This kind of a...Yes, it's all derivative free.<br /><br />This kind of algorithm only works for problems with a relatively small number of parameters. There is no way it could possibly optimize a problem with 10s of thousands of parameters (or more, some DNNs have millions to billions of parameters). This is true for any derivative free algorithm. It's just not going to work.<br /><br />But if you have something like 5 parameters you want to optimize then it's great.Davis Kinghttps://www.blogger.com/profile/16577392965630448489noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-83472258503272970612018-05-17T04:35:53.921-04:002018-05-17T04:35:53.921-04:00Hi Davis
Apologies in advance if some of these is...Hi Davis<br /><br />Apologies in advance if some of these is very basic.<br /><br />RE:<br />"You could combine the trust region strategy with pretty much any other derivative free solver."<br />Just to be 100% clear: "the trust region strategy" is *itself* derivative-free (looking at BOBYQA definition) - is my understanding right?<br /><br /><br />RE:<br />Q: "Is it possible to use this brilliant optimizer in the training of neural network of machine learning? Is this optimzer is something that could replace traditional optimizer like Adam,AdaGrad...?"<br />A: "No, it's not going to be reasonable to use this in place of adam or sgd."<br /><br />Is this true only of neural network hyperparam optimisations, or more in genearl of machine learning hyperparam optimisation problems? <br />Would you mind expanding on this?<br /><br />Reiterating my gratitude for your excellent work.Mario Riverahttps://www.blogger.com/profile/10835086690809997056noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-86823581733882319642018-05-16T11:07:35.170-04:002018-05-16T11:07:35.170-04:00No, it's not going to be reasonable to use thi...No, it's not going to be reasonable to use this in place of adam or sgd.Davis Kinghttps://www.blogger.com/profile/16577392965630448489noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-73183556933442176722018-05-16T09:58:15.198-04:002018-05-16T09:58:15.198-04:00Is it possible to use this brilliant optimizer in ...Is it possible to use this brilliant optimizer in the training of neural network of machine learning? Is this optimzer is something that could replace traditional optimizer like Adam,AdaGrad...?Junwei Donghttps://www.blogger.com/profile/05066597905219881172noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-60839874572030610332018-05-15T07:08:02.946-04:002018-05-15T07:08:02.946-04:00You could combine the trust region strategy with p...You could combine the trust region strategy with pretty much any other derivative free solver. I used LIPO here because I find LIPO the most compelling. I have not compared it to HORD. Davis Kinghttps://www.blogger.com/profile/16577392965630448489noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-75865331939487111642018-05-15T03:50:15.331-04:002018-05-15T03:50:15.331-04:00Hi
Excellent work, thank you for sharing.
Two que...Hi<br /><br />Excellent work, thank you for sharing.<br />Two questions:<br />1) Can this be combined with hyperband https://archive.is/CDVR8 ?<br />2) How does this compare to HORD / pysot (https://archive.is/YgRvr)? Do you have any comment on that approach?<br /><br />Greetings from London<br />Mario Riverahttps://www.blogger.com/profile/10835086690809997056noreply@blogger.comtag:blogger.com,1999:blog-6061887630060661987.post-80577189620255065472018-05-02T11:16:34.637-04:002018-05-02T11:16:34.637-04:00Thanks, glad you like it.
Yes, you can do this. ...Thanks, glad you like it.<br /><br />Yes, you can do this. See the extended discussion here: http://dlib.net/dlib/global_optimization/global_function_search_abstract.h.html#global_function_search. That's the C++ interface. There is a python interface with essentially an identical API as well.Davis Kinghttps://www.blogger.com/profile/16577392965630448489noreply@blogger.com