tag:blogger.com,1999:blog-60618876300606619872024-03-19T06:12:34.353-04:00dlib C++ LibraryDavis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.comBlogger22125tag:blogger.com,1999:blog-6061887630060661987.post-82457861368203374342018-02-13T08:02:00.000-05:002018-02-13T08:02:33.516-05:00Automatic Learning Rate Scheduling That Really WorksTraining deep learning models can be a pain. In particular, there is this perception that one of the reasons it's a pain is because you have to fiddle with learning rates. For example, arguably the most popular strategy for setting learning rates looks like this:<br />
<ol>
<li>Run vanilla stochastic gradient descent with momentum and a fixed learning rate</li>
<li>Wait for the loss to stop improving</li>
<li>Reduce the learning rate</li>
<li>Go back to step 1 or stop if the learning rate is really small</li>
</ol>
<div>
Many papers reporting state-of-the-art results do this. There have been a lot of other methods proposed, like ADAM, but I've always found the above procedure to work best. <a href="https://arxiv.org/abs/1705.08292">This is a common finding</a>. The only fiddly part of this procedure is the "wait for the loss to stop improving" step. A lot of people just eyeball a plot of the loss and manually intervene when it looks like its flattened out. Or worse, they pick a certain number of iterations ahead of time and blindly stop when that limit is reached. Both of these ways of deciding when to reduce the learning rate suck. </div>
<br />
Fortunately, there is a simple method from classical statistics we can use to decide if the loss is still improving, and thus, when to reduce it. With this method it's trivial to fully automate the above procedure. In fact, it's what I've used to train all the public DNN models in dlib over the last few years: e.g. <a href="http://blog.dlib.net/2016/10/easily-create-high-quality-object.html">face detection</a>, <a href="http://blog.dlib.net/2017/02/high-quality-face-recognition-with-deep.html">face recognition</a>, <a href="http://blog.dlib.net/2017/08/vehicle-detection-with-dlib-195_27.html">vehicle detection</a>, and <a href="http://blog.dlib.net/2016/06/a-clean-c11-deep-learning-api.html">imagenet classification</a>. It's the default solving strategy used by <a href="http://dlib.net/ml.html#dnn_trainer">dlib's DNN solver</a>. The rest of this blog post explains how it works.<br />
<br />
Fundamentally, what we need is a method that takes a noisy time series of $n$ loss values, $Y=\{y_0,y_1,y_2,\dots,y_{n-1}\}$, and tells us if the time series is trending down or not. To do this, we model the time series as a noisy line corrupted by Gaussian noise:<br />
\[<br />
\newcommand{\N} {\mathcal{N} } y_i = m\times i + b + \epsilon<br />
\] Here, $m$ and $b$ are the unknown true slope and intercept parameters of the line, and $\epsilon$ is a Gaussian noise term with mean 0 and variance $\sigma^2$. Let's also define the function $\text{slope}(Y)$ that takes in a time series, performs <a href="https://en.wikipedia.org/wiki/Ordinary_least_squares">OLS</a>, and outputs the OLS estimate of $m$, the slope of the line. You can then ask the following question: what is the probability that a time series sampled from our noisy line model will have a negative slope according to OLS? That is, what is the value of?<br />
\[<br />
P(\text{slope}(Y) < 0)<br />
\]If we could compute an estimate of $P(\text{slope}(Y)<0)$ we could use it to test if the loss is still decreasing. Fortunately, computing the above quantity turns out to be easy. In fact, $\text{slope}(Y)$ is a Gaussian random variable with this distribution:<br />
\[<br />
\text{slope}(Y) \sim \N\left(m, \frac{12 \sigma^2}{n^3-n}\right)<br />
\]We don't know the true values of $m$ and $\sigma^2$, but they are easily estimated from data. We can obviously use $\text{slope}(Y)$ to estimate $m$. As for $\sigma^2$, it's customary to estimate it like this:<br />
\[ \sigma^2 = \frac{1}{n-2} \sum_{i=0}^{n-1} (y_i - \hat y_i)^2 \] which gives an unbiased estimate of the true $\sigma^2$. Here $y_i - \hat y_i$ is the difference between the observed time series value at time $i$ and the value predicted by the OLS fitted line at time $i$. I should point out that none of this is new stuff, in fact, these properties of OLS are discussed in detail on the <a href="https://en.wikipedia.org/wiki/Ordinary_least_squares#Finite_sample_properties">Wikipedia page about OLS</a>.<br />
<br />
So let's recap. We need a method to decide if the loss is trending downward or not. I'm suggesting that you use $P(\text{slope}(Y) < 0)$, the probability that a line fit to your loss curve will have negative slope. Moreover, as discussed above, this probability is easy to compute since it's just a question about a simple Gaussian variable and the two parameters of the Gaussian variable are given by a straightforward application of OLS.<br />
<br />
You should also note that the variance of $\text{slope}(Y)$ decays at the very quick rate of $O(1/n^3)$, where $n$ is the number of loss samples. So it becomes very accurate as the length of the time series grows. To illustrate just how accurate this is, let's look at some examples. The figure below shows four different time series plots, each consisting of $n=4000$ points. Each plot is a draw from our noisy line model with parameters: $b=0$, $\sigma^2=1$, and $m \in \{0.001, 0.00004, -0.00004, -0.001\}$. For each of these noisy plots I've computed $P(\text{slope}(Y) < 0)$ and included it in the title.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGn1z2KcdTvHFDrHIWPnjr3zc27s3P3Gt77xjQi9qsurll6sa027asm_Gu4HO47-VvKVGwN19xVu7Hh9asHNX7sgYOxQVW0KmDS7HHutU17VMt4Sll9fnuNNEsLLUnBxx2lijPypQ2i8c/s1600/noisy_lines2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGn1z2KcdTvHFDrHIWPnjr3zc27s3P3Gt77xjQi9qsurll6sa027asm_Gu4HO47-VvKVGwN19xVu7Hh9asHNX7sgYOxQVW0KmDS7HHutU17VMt4Sll9fnuNNEsLLUnBxx2lijPypQ2i8c/s1600/noisy_lines2.png" style="all: initial;" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
From looking at these plots it should be obvious that $P(\text{slope}(Y) < 0)$ is quite good at detecting the slope. In particular, I doubt you can tell the difference between the two middle plots (the ones with slopes -0.00004 and 0.00004). But as you can see, the test statistic I'm suggesting, $P(\text{slope}(Y) < 0)$, has no trouble at all correctly identifying one as sloping up and the other as sloping down.<br />
<br />
I find that a nice way to parameterize this in actual code is to count the number of mini-batches that executed while $P(\text{slope}(Y) < 0) < 0.51$. That is, find out how many loss values you have to look at before there is evidence the loss has been decreasing. To be very clear, this bit of pseudo-code implements the idea:<br />
<div class="highlight">
<pre> <span class="k">def</span> <span class="nf">count_steps_without_decrease</span><span class="p">(</span><span class="n">Y</span><span class="p">):</span>
<span class="n">steps_without_decrease</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">n</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">Y</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">reversed</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">)):</span>
<span class="k">if</span> <span class="n">P</span><span class="p">(</span><span class="n">slope</span><span class="p">(</span><span class="n">Y</span><span class="p">[</span><span class="n">i</span><span class="p">:</span><span class="n">n</span><span class="p">])</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="o"><</span> <span class="mf">0.51</span><span class="p">:</span>
<span class="n">steps_without_decrease</span> <span class="o">=</span> <span class="n">n</span><span class="o">-</span><span class="n">i</span>
<span class="k">return</span> <span class="n">steps_without_decrease</span>
</pre>
</div>
You can then use a rule like: "if the steps without decrease is 1000 I will lower the learning rate by 10x". However, there is one more issue that needs to be addressed. This is the fact that loss curves sometimes have really large transient spikes, where, for one reason or another (e.g. maybe a bad mini-batch) the loss will suddenly become huge for a moment. Not all models or datasets have this problem during training, but some do. In these cases, count_steps_without_decrease() might erroneously return a very large value. You can deal with this problem by discarding the top 10% of loss values inside count_steps_without_decrease(). This makes the entire procedure robust to these noisy outliers. Note, however, that the final test you would want to use is:<br />
<pre><span class="nf">count_steps_without_decrease(Y) > threshold and </span>count_steps_without_decrease_robust(Y) > threshold</pre>
That is, perform the check with and without outlier discarding. You need both checks because the 10% largest loss values might have occurred at the very beginning of Y. For example, maybe you are waiting for 1000 (i.e. threshold=1000) mini-batches to execute without showing evidence of the loss going down. And maybe the first 100 all showed a dropping loss while the last 900 were flat. The check that discarded the top 10% would erroneously indicate that the loss was NOT dropping. So you want to perform both checks and if both agree that the loss isn't dropping then you can be confident it's not dropping.<br />
<br />
It should be emphasized that this method isn't substantively different from what a whole lot of people already do when training deep neural networks. The only difference here is that the "look at the loss and see if it's decreasing" step is being done by a computer. The point of this blog post is to point out that this check is trivially automatable with boring old simple statistics. There is no reason to do it by hand. Let the computer do it and find something more productive to do with your time than babysitting SGD. The test is simple to implement yourself, but if you want to just call a function you can call dlib's <a href="http://dlib.net/algorithms.html#count_steps_without_decrease">count_steps_without_decrease</a>() and <a href="http://dlib.net/algorithms.html#count_steps_without_decrease_robust">count_steps_without_decrease_robust</a>() routines from C++ or Python.<br />
<br />
Finally, one more useful thing you can do is the following: you can periodically check if $P(\text{slope}(Y) > 0) \gt 0.99$, that is, check if we are really certain that the loss is going up, rather than down. This can happen and I've had training runs that were going fine and then suddenly the loss shot up and stayed high for a really long time, basically ruining the training run. This doesn't seem to be too much of an issue with simple losses like the log-loss. However, structured loss functions that perform some kind of hard negative mining inside a mini-batch will sometimes go haywire if they hit a very bad mini-batch. You can fix this problem by simply reloading from an earlier network state before the loss increased. But to do this you need a reliable way to measure "the loss is going up" and $P(\text{slope}(Y) > 0) \gt 0.99$ is excellent for this task. This idea is called backtracking and has a long history in numerical optimization. Backtracking significantly increases solver robustness in many cases and is well worth using.Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com35tag:blogger.com,1999:blog-6061887630060661987.post-55731656026320503862018-01-14T11:09:00.001-05:002018-01-14T11:20:31.497-05:00Correctly Mirroring DatasetsI get asked a lot of questions about <a href="http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html">dlib's landmarking tools</a>. Some of the most common questions are about how to prepare a good training dataset. One of the most useful tricks for creating a dataset is to mirror the data, since this effectively doubles the amount of training data. However, if you do this naively you end up with a terrible training dataset that produces really awful landmarking models. Some of the most common questions I get are about why this is happening.<br />
<br />
To understand the issue, consider the following image of an annotated face from the <a href="https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/">iBug W-300 dataset</a>:<br />
<br />
<div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgd-DDJj0G4nM_1nUYlI_9LBJxso8t6TsjZYprfodAEHZU141BAhzOReCj7_tJAPxv9-VJdldH096blmxtt6xGAOsRJO5_r4YyXfI3MzE6GtF7OO8j1Vjikkt8fvdJ_vz1Kx2aAvHC5WAI/s1600/Screenshot+from+2018-01-14+09-43-26.png" imageanchor="1"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgd-DDJj0G4nM_1nUYlI_9LBJxso8t6TsjZYprfodAEHZU141BAhzOReCj7_tJAPxv9-VJdldH096blmxtt6xGAOsRJO5_r4YyXfI3MzE6GtF7OO8j1Vjikkt8fvdJ_vz1Kx2aAvHC5WAI/s1600/Screenshot+from+2018-01-14+09-43-26.png" /></a></div>
<div>
<br /></div>
<div>
Since the mirror image of a face is still a face, we can mirror images like this to get more training data. However, what happens if you simply mirror the annotations? You end up with the wrong annotation labels! To see this, take a look at the figure below. The left image shows what happens if you naively mirror the above image and its landmarks. Note, for instance, that the points along the jawline are now annotated in reverse order. In fact, nearly all the annotations in the left image are wrong. Instead, <b>you want to match the source image's labeling scheme</b>. A mirrored image with the correct annotations is shown on the right.</div>
<div>
<br /></div>
<div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYOC89K9xj3SD6g2excgDUzVSSLLp32SGtbjxaA_IhqM5BH37xzZHnDK2SzvjkTrWxe1dfu92oWsrRFR3n-OJOI-HT6gZDgDNUctW49e9B8NdQjX3qx_f2OhooUUa-vP5T6qpblIEQuLA/s1600/simple_mirroring_vs_source_label_matching.png" imageanchor="1"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYOC89K9xj3SD6g2excgDUzVSSLLp32SGtbjxaA_IhqM5BH37xzZHnDK2SzvjkTrWxe1dfu92oWsrRFR3n-OJOI-HT6gZDgDNUctW49e9B8NdQjX3qx_f2OhooUUa-vP5T6qpblIEQuLA/s1600/simple_mirroring_vs_source_label_matching.png" /></a></div>
<div>
<br /></div>
<div>
Dlib's <a href="https://github.com/davisking/dlib/tree/master/tools/imglab">imglab</a> tool has had a --flip option for a long time that would mirror a dataset for you. However, it used naive mirroring and it was left up to the user to adjust any landmark labels appropriately. Many users found this confusing, so in the new version of imglab (v1.13) the --flip command now performs automatic source label matching using a <a href="https://en.wikipedia.org/wiki/Point_set_registration">2D point registration algorithm</a>. That is, it left-right flips the dataset and annotations. Then it registers the mirrored landmarks with the original landmarks and transfers labels appropriately. In fact, the "source label matching" image on the right was created by the new version of imglab.<br />
<br />
Finally, just to be clear, the point registration algorithm will work on anything. It doesn't have to be iBug's annotations. It doesn't have to be faces. It's a general point registration method that will work correctly for any kind of landmark annotated data with left-right symmetry. However, if you want the old --flip behavior you can use the new --flip-basic to get a naive mirroring. But most users will want to use the new --flip.</div>
Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com20tag:blogger.com,1999:blog-6061887630060661987.post-40763411702685764832017-12-28T11:09:00.002-05:002018-01-01T15:23:31.635-05:00A Global Optimization Algorithm Worth UsingHere is a common problem: you have some machine learning algorithm you want to use but it has these damn <i>hyperparameters</i>. These are numbers like weight decay magnitude, Gaussian kernel width, and so forth. The algorithm doesn't set them, instead, it's up to you to determine their values. If you don't set these parameters to "good" values the algorithm doesn't work. So what do you do? Well, here is a list of everything I've seen people do, listed in order of most to least common:<br />
<ul>
<li><b>Guess and Check</b>: Listen to your gut, pick numbers that feel good and see if they work. Keep doing this until you are tired of doing it.</li>
<li><b>Grid Search</b>: Ask your computer to try a bunch of values spread evenly over some range.</li>
<li><b>Random Search</b>: Ask your computer to try a bunch of values by picking them randomly. </li>
<li><b>Bayesian Optimization</b>: Use a tool like MATLAB's <a href="https://www.mathworks.com/help/stats/bayesopt.html">bayesopt</a> to automatically pick the best parameters, then find out Bayesian Optimization has more hyperparameters than your machine learning algorithm, get frustrated, and go back to using guess and check or grid search.</li>
<li><b>Local Optimization With a Good Initial Guess</b>: This is what <a href="https://github.com/mit-nlp/MITIE">MITIE</a> does, it uses the <a href="http://dlib.net/optimization.html#find_min_bobyqa">BOBYQA</a> algorithm with a well chosen starting point. Since BOBYQA only finds the nearest <i>local</i> optima the success of this method is heavily dependent on a good starting point. In MITIE's case we know a good starting point, but this isn't a general solution since usually you won't know a good starting point. On the plus side, this kind of method is extremely good at finding a local optima. I'll have more to say on this later.</li>
</ul>
<div>
The vast majority of people just do guess and check. That sucks and there should be something better. We all want some black-box optimization strategy like Bayesian optimization to be useful, but in my experience, if you don't set its hyperparameters to the right values it doesn't work as well as an expert doing guess and check. Everyone I know who has used Bayesian optimization has had the same experience. Ultimately, if I think I can do better hyperparameter selection manually then that's what I'm going to do, and most of my colleagues feel the same way. The end result is that I don't use automated hyperparameter selection tools most of the time, and that bums me out. I badly want a parameter-free global optimizer that I can trust to do hyperparameter selection.<br />
<br />
So I was very excited when I encountered the paper <i><a href="https://arxiv.org/abs/1703.02628">Global optimization of Lipschitz functions</a></i> by Cédric Malherbe and Nicolas Vayatis in this year's international conference on machine learning. In this paper, they propose a very simple <i>parameter-free and provably correct method</i> for finding the $x \in \mathbb{R}^d$ that maximizes a function, $f(x)$, even if $f(x)$ has many local maxima. The key idea in their paper is to maintain a piecewise linear upper bound of $f(x)$ and use that to decide which $x$ to evaluate at each step of the optimization. So if you already evaluated the points $x_1, x_2, \cdots, x_t$ then you can define a simple upper bound on $f(x)$ like this:<br />
\[ \newcommand{\norm}[1]{\left\lVert#1\right\rVert} U(x) = \min_{i=1\dots t} (f(x_i) + k \cdot \norm{x-x_i}_2 ) \] Where $k$ is the Lipschitz constant for $f(x)$. Therefore, it is trivially true that $U(x) \geq f(x), \forall x$, by the definition of the Lipschitz constant. The authors go on to suggest a simple algorithm, called LIPO, that picks points at random, checks if the upper bound for the new point is better than the best point seen so far, and if so selects it as the next point to evaluate. For example, the figure below shows a plot of a simple $f(x)$ in red with a plot of its associated upper bound $U(x)$ in green. In this case $U(x)$ is defined by 4 points, indicated here with little black squares.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbxnvbpzx69SDM0zVbgtxzL1cZ4BHxJTu1wFjwsArfCWx4_oWK1CkoNUdPfvJVt89hE3NRwAbDaOsBcND0tip6u9EMqP6ptnp0YAvusQKrb7PvwhyphenhyphenSkoEKcWt3sAUoBNcX0ZfSqcOfs3o/s1600/g4175.png" style="all: initial;" /></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
It shouldn't take a lot of imagination to see how the upper bound helps you pick good points to evaluate. For instance, if you selected the max upper bound as the next iterate you would already get pretty close to the global maximizer. The authors go on to prove a bunch of nice properties of this method. In particular, they both prove mathematically and show empirically that the method is better than random search in a number of non-trivial situations. This is a fairly strong statement considering how competitive <a href="http://www.jmlr.org/papers/v13/bergstra12a.html">random hyperparameter search</a> turns out to be relative to competing hyperparameter optimization methods. They also compare the method to other algorithms like Bayesian optimization and show that it's competitive.<br />
<br />
But you are probably thinking: "Hold on a second, we don't know the value of the Lipschitz constant $k$!". This isn't a big deal since it's easily estimated, for instance, by setting $k$ to the largest observed slope of $f(x)$ before each iteration. That's equivalent to solving the following easy problem:<br />
\begin{align}<br />
\min_{k} & \quad k^2 \\<br />
\text{s.t.} & \quad U(x_i) \geq f(x_i), \quad \forall i \in [1\dots t] \\<br />
& \quad k \geq 0<br />
\end{align} Malherbe et al. test a variant of this $k$ estimation approach and show it works well. <br />
<br />
This is great. I love this paper. It's proposing a global optimization method called LIPO that is both parameter free and provably better than random search. It's also really simple. Reading this paper gives you one of those "duah" moments where you wonder why you didn't think of this a long time ago. That's the mark of a great paper. So obviously I was going to add some kind of LIPO algorithm to dlib, which I did in the recent <a href="http://dlib.net/release_notes.html">dlib v19.8 release</a>.<br />
<br />
However, if you want to use LIPO in practice there are some issues that need to be addressed. The rest of this blog post discusses these issues and how the dlib implementation addresses them. First, if $f(x)$ is noisy or discontinuous even a little it's not going to work reliably since $k$ will be infinity. This happens in real world situations all the time. For instance, evaluating a binary classifier against the 0-1 loss gives you an objective function with little discontinuities anywhere samples switch their predicted class. You could cross your fingers and run LIPO anyway, but you run the very real risk of two $x$ samples closely straddling a discontinuity and causing the estimated $k$ to explode. Second, not all hyperparameters are equally important, some hardly matter while small changes in others drastically affect the output of $f(x)$. So it would be nice if each hyperparameter got its own $k$. You can address these problems by defining the upper bound $U(x)$ as follows:<br />
\[ U(x) = \min_{i=1\dots t} \left[ f(x_i) + \sqrt{\sigma_i +(x-x_i)^\intercal K (x-x_i)} \ \right] \] Now each sample from $f(x)$ has its own noise term, $\sigma_i$, which should be 0 most of the time unless $x_i$ is really close to a discontinuity or there is some stochasticity. Here, $K$ is a diagonal matrix that contains our "per hyperparameter Lipschitz $k$ terms". With this formulation, setting each $\sigma$ to 0 and $K=k^2I$ gives the same $U(x)$ as suggested by Malherbe et al., but if we let them take more general values we can deal with the above mentioned problems.<br />
<br />
Just like before, we can find the parameters of $U(x)$ by solving an optimization problem:<br />
\begin{align}<br />
\min_{K,\sigma} & \quad \norm{K}^2_F + 10^6 \sum_{i=1}^t {\sigma_i^2} &\\<br />
\text{s.t.} & \quad U(x_i) \geq f(x_i), & \quad \forall i \in [1\dots t] \\<br />
& \quad \sigma_i \geq 0 & \quad \forall i \in [1\dots t] \\<br />
& \quad K_{i,j} \geq 0 & \quad \forall i,j \in [1\dots d] \\<br />
& \quad \text{K is a diagonal matrix}<br />
\end{align} The $10^6$ penalty on $\sigma^2$ causes most $\sigma$ terms to be exactly 0. The behavior of the whole algorithm is insensitive to the particular penalty value used here, so long as it's reasonably large the $\sigma$ values will be 0 most of the time while still preventing $k$ from becoming infinite, which is the behavior we want. It's also possible to rewrite this as a big quadratic programming problem and solve it with a dual coordinate descent method. I'm not going into the details here. It's all in the dlib code for those really interested. The TL;DR is that it turns out to be easy to solve using well known methods and it fixes the infinite $k$ problem.<br />
<br />
The final issue that needs to be addressed is LIPO's terrible convergence in the area of a local maximizer. So while it's true that LIPO is great at getting onto the tallest peak of $f(x)$, once you are there it does not make very rapid progress towards the optimal location (i.e. the very top of the peak). This is a problem shared by many <a href="https://en.wikipedia.org/wiki/Derivative-free_optimization">derivative free optimization</a> algorithms, including MATLAB's Bayesian optimization tool. Fortunately, not all methods have this limitation. In particular, the late and great <a href="https://en.wikipedia.org/wiki/Michael_J._D._Powell">Michael J. D. Powell</a> wrote a series of papers on how to apply classic trust region methods to derivative free optimization. These methods fit a quadratic surface around the best point seen so far and then take the next iterate to be the maximizer of that quadratic surface within some distance of the current best point. So we "trust" this local quadratic model to be accurate within some small region around the best point, hence the name "trust region". The BOBYQA method I mentioned above is one of these methods and it has excellent convergence to the nearest local optima, easily finding local optima to full floating point precision in a very small number of steps.<br />
<br />
We can fix LIPO's convergence problem by combining these two methods, LIPO will explore $f(x)$ and quickly find a point on the biggest peak. Then a Powell style trust region method can efficiently find the exact maximizer of that peak. The simplest way to combine these two things is to alternate between them, which is what dlib does. On even iterations we pick the next $x$ according to our upper bound while on odd iterations we pick the next $x$ according to the trust region model. I've also used a slightly different version of LIPO that I'm calling MaxLIPO. Recall that Malherbe et al. suggest selecting any point with an upper bound larger than the current best objective value. However, I've found that selecting the maximum upper bounding point on each iteration is slightly better. This alternative version, MaxLIPO, is therefore what dlib uses. You can see this hybrid of MaxLIPO and a trust region method in action in the following video:<br />
<br />
<center>
<video controls="true" poster="http://dlib.net/find_max_global_example.png" style="border: black dotted 1px;" width="80%">
<source src="http://dlib.net/find_max_global_example.webm" type="video/webm"></source>
<source src="http://dlib.net/find_max_global_example.mp4" type="video/mp4"></source>
Video of optimizer running
</video></center>
<br />
In the video, the red line is the function to be optimized and we are looking for the maximum point. Every time the algorithm samples a point from the function we note it with a little box. The state of the solver is determined by the global upper bound $U(x)$ and the local quadratic model used by the trust region method. Therefore, we draw the upper bounding model as well as the current local quadratic model so you can see how they evolve as the optimization proceeds. We also note the location of the best point seen so far by a little vertical line.<br />
<br />
You can see that the optimizer is alternating between picking the maximum upper bounding point and the maximum point according to the quadratic model. As the optimization proceeds, the upper bound becomes progressively more accurate, helping to find the best peak to investigate, while the quadratic model quickly finds a high precision maximizer on whatever peak it currently rests. These two things together allow the optimizer to find the true global maximizer to high precision (within $\pm{10^{-9}}$ in this case) by the time the video concludes.</div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiflNDIB8ATDOxLqjR308fmEJiX2w1fBB5wNXPKjAEfcLBVfPSzAqk2DCNHSMPLDxj4ECpmTpGB-NWtL7VmLI7uiYOegCT4_xyl5ppzA5pAfeZtiGyU1w8bhDIevnQ8cbQ2mAVyVlm5esA/s1600/page1-800px-Holder_table_function.pdf.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiflNDIB8ATDOxLqjR308fmEJiX2w1fBB5wNXPKjAEfcLBVfPSzAqk2DCNHSMPLDxj4ECpmTpGB-NWtL7VmLI7uiYOegCT4_xyl5ppzA5pAfeZtiGyU1w8bhDIevnQ8cbQ2mAVyVlm5esA/s400/page1-800px-Holder_table_function.pdf.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b><span style="font-size: small;">The Holder Table Test Function</span></b><br />
<span style="font-size: xx-small;">from https://en.wikipedia.org/wiki/File:Holder_table_function.pdf</span></td></tr>
</tbody></table>
<br />
Now let's do an experiment to see how this hybrid of MaxLIPO and Powell's trust region method (TR) compares to MATLAB's Bayesian optimization tool with its default settings. I ran both algorithms on the Holder table test function 100 times and plotted the average error with one standard deviation error bars. So the plot below shows $f(x^\star)-f(x_i)$, the difference between the true global optimum and the best solution found so far, as a function of the number of calls to $f(x)$. You can see that MATLAB's BayesOpt stalls out at an accuracy of about $\pm{10^{-3}}$ while our hybrid method (MaxLIPO+TR, the new method in dlib) quickly approaches full floating point precision of around $\pm{10^{-17}}$.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHrAke1weMeC2GlVb6fN9ktraWJPuOV4N3l0mOqsuaio7uqUVHT6oqi2bJHWpJtGbwvwzyRe7QBFSAl4WyjgYBOBuUgu8sTnpIub6RL3QeISKr1R6AfxOPn0YUEBvCI8DTJTBC7MiD7cE/s1600/bayesopt_vs_lipo.svg.png" style="all: initial; all: unset;" /> </div>
<br />
I also reran some of the tests from Figure 5 of the LIPO paper. The results are shown in the table below. In these experiments I compared the performance of LIPO with and without the trust region solver (LIPO+TR and LIPO). Additionally, to verify that LIPO is better than pure random search I tested a version of the algorithm that alternates between pure random search and the trust region solver (PRS+TR) rather than alternating between a LIPO method and a trust region solver (LIPO+TR and MaxLIPO+TR). Pure random search (PRS) is also included for reference. Finally, the new algorithm implemented in dlib, MaxLIPO+TR, is included as well. In each test I ran the algorithm 1000 times and recorded the mean and standard deviation of the number of calls to $f(x)$ required to reach a particular solution accuracy. For instance, $\epsilon=0.01$ means that $f(x^\star)-f(x_i) \leq 0.01$, while "target 99%" uses the "target" metric from Malherbe's paper, which for most tests corresponds to an $\epsilon > 0.1$. Tests that took too long to execute are noted with a - symbol.<br />
<br />
The key points to notice about these results are that the addition of a trust region method allows LIPO to reach much higher solution accuracy. It also makes the algorithm run faster. Recall that LIPO works internally by using random search of $U(x)$. Therefore, the number of calls LIPO makes to $U(x)$ is at least as many as PRS would require when searching $f(x)$. So for smaller $\epsilon$ it becomes very expensive to execute LIPO. For instance, I wasn't able to get results for LIPO, by itself, at accuracies better than $0.1$ on any of the test problems since it took too long to execute. However, with a trust region method the combined algorithm can easily achieve high precision solutions. The other significant detail is that, for tests with many local optima, all methods combining LIPO with TR are much better than PRS+TR. This is most striking on ComplexHolder, which is a version of the HolderTable test function with additional high frequency sinusoidal noise that significantly increases the number of local optima. On ComplexHolder, LIPO based methods require about an order of magnitude fewer calls to $f(x)$ than PRS+TR, further justifying the claims by Malherbe et al. of the superiority of LIPO relative to pure random search.<br />
<br />
<center>
<img src="http://dlib.net/find_max_global_results_table.svg" width="100%" /></center>
<br />
The new method in dlib, MaxLIPO+TR, fares the best in all my tests. What is remarkable about this method is its simplicity. In particular, <b>MaxLIPO+TR doesn't have any hyperparameters, making it very easy to use.</b> I've been using it for a while now for hyperparameter optimization and have been very pleased. It's the first black-box hyperparameter optimization algorithm I've had enough confidence in to use on real problems.<br />
<br />
Finally, here is an example of how you can use this new optimizer from Python:<br />
<pre style="background-color: white; line-height: 16.25px;"><span class="k" style="color: blue;">def</span> <span class="nf">holder_table</span><span class="p">(</span><span class="n">x0</span><span class="p">,</span><span class="n">x1</span><span class="p">):</span>
<span class="k" style="color: blue;">return</span> <span class="o">-</span><span class="nb">abs</span><span class="p">(</span><span class="n">sin</span><span class="p">(</span><span class="n">x0</span><span class="p">)</span><span class="o">*</span><span class="n">cos</span><span class="p">(</span><span class="n">x1</span><span class="p">)</span><span class="o">*</span><span class="n">exp</span><span class="p">(</span><span class="nb">abs</span><span class="p">(</span><span class="mi">1</span><span class="o">-</span><span class="n">sqrt</span><span class="p">(</span><span class="n">x0</span><span class="o">*</span><span class="n">x0</span><span class="o">+</span><span class="n">x1</span><span class="o">*</span><span class="n">x1</span><span class="p">)</span><span class="o">/</span><span class="n">pi</span><span class="p">)))</span>
<span class="n">x</span><span class="p">,</span><span class="n">y</span> <span class="o">=</span> <span class="n">dlib</span><span class="o">.</span><span class="n">find_min_global</span><span class="p">(</span><span class="n">holder_table</span><span class="p">,</span>
<span class="p">[</span><span class="o">-</span><span class="mi">10</span><span class="p">,</span><span class="o">-</span><span class="mi">10</span><span class="p">],</span> <span class="c1" style="color: green;"># Lower bound constraints on x0 and x1 respectively</span>
<span class="p">[</span><span class="mi">10</span><span class="p">,</span><span class="mi">10</span><span class="p">],</span> <span class="c1" style="color: green;"># Upper bound constraints on x0 and x1 respectively</span>
<span class="mi">80</span><span class="p">)</span> <span class="c1" style="color: green;"># The number of times find_min_global() will call holder_table()</span></pre>
<pre style="background-color: white; line-height: 16.25px;"><span class="c1" style="color: green;">
</span></pre>
<pre style="background-color: white; line-height: 16.25px;">Or in C++11:</pre>
<pre><span style="color: blue;">auto</span> holder_table <span style="color: #5555ff;">=</span> []<span style="font-family: "lucida console";">(</span><span style="color: blue;"><u>double</u></span> x0, <span style="color: blue;"><u>double</u></span> x1<span style="font-family: "lucida console";">)</span> <b>{</b><span style="color: blue;">return</span> <span style="color: #5555ff;">-</span><span style="color: #bb00bb;">abs</span><span style="font-family: "lucida console";">(</span><span style="color: #bb00bb;">sin</span><span style="font-family: "lucida console";">(</span>x0<span style="font-family: "lucida console";">)</span><span style="color: #5555ff;">*</span><span style="color: #bb00bb;">cos</span><span style="font-family: "lucida console";">(</span>x1<span style="font-family: "lucida console";">)</span><span style="color: #5555ff;">*</span><span style="color: #bb00bb;">exp</span><span style="font-family: "lucida console";">(</span><span style="color: #bb00bb;">abs</span><span style="font-family: "lucida console";">(</span><span style="color: #979000;">1</span><span style="color: #5555ff;">-</span><span style="color: #bb00bb;">sqrt</span><span style="font-family: "lucida console";">(</span>x0<span style="color: #5555ff;">*</span>x0<span style="color: #5555ff;">+</span>x1<span style="color: #5555ff;">*</span>x1<span style="font-family: "lucida console";">)</span><span style="color: #5555ff;">/</span>pi<span style="font-family: "lucida console";">)</span><span style="font-family: "lucida console";">)</span><span style="font-family: "lucida console";">)</span>;<b>}</b>;
<span style="color: #009900;">// obtain result.x and result.y
</span><span style="color: blue;">auto</span> result <span style="color: #5555ff;">=</span> <span style="color: #bb00bb;">find_min_global</span><span style="font-family: "lucida console";">(</span>holder_table,
<b>{</b><span style="color: #5555ff;">-</span><span style="color: #979000;">10</span>,<span style="color: #5555ff;">-</span><span style="color: #979000;">10</span><b>}</b>, <span style="color: #009900;">// lower bounds
</span> <b>{</b><span style="color: #979000;">10</span>,<span style="color: #979000;">10</span><b>}</b>, <span style="color: #009900;">// upper bounds
</span> <span style="color: #bb00bb;">max_function_calls</span><span style="font-family: "lucida console";">(</span><span style="color: #979000;">80</span><span style="font-family: "lucida console";">)</span><span style="font-family: "lucida console";">)</span>;</pre>
<br />
Both of these methods find holder_table's global optima to about 12 digits of precision in about 0.1 seconds. The C++ API exposes a wide range of ways to call the solver, including optimizing multiple functions at a time and adding integer constraints. <a href="http://dlib.net/optimization.html#find_max_global">See the documentation for full details</a>.<br />
<br />Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com102tag:blogger.com,1999:blog-6061887630060661987.post-55949499463669196342017-12-20T06:27:00.000-05:002018-01-14T19:32:20.619-05:00Dlib 19.8 is OutDlib 19.8 is officially out. There are <a href="http://dlib.net/release_notes.html">a lot of changes</a>, but the two most interesting ones are probably the <a href="http://dlib.net/global_optimization.py.html">new global optimizer</a> and <a href="http://dlib.net/dnn_semantic_segmentation_ex.cpp.html">semantic segmentation</a> examples. The global optimizer is definitely my favorite as it allows you to easily find the optimal hyperparameters for machine learning algorithms. It also has a very convenient syntax. For example, consider the Holder table test function:<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><img alt="File:Holder table function.pdf" height="300" src="https://upload.wikimedia.org/wikipedia/commons/thumb/7/72/Holder_table_function.pdf/page1-800px-Holder_table_function.pdf.jpg" style="margin-left: auto; margin-right: auto;" width="400" /></td></tr>
<tr><td class="tr-caption" style="text-align: center;">From https://en.wikipedia.org/wiki/File:Holder_table_function.pdf</td></tr>
</tbody></table>
Here is how you could use dlib's new optimizer from Python to optimize the difficult Holder table function:<br />
<pre style="background-color: white; line-height: 16.25px;"><span class="k" style="color: blue;">def</span> <span class="nf">holder_table</span><span class="p">(</span><span class="n">x0</span><span class="p">,</span><span class="n">x1</span><span class="p">):</span>
<span class="k" style="color: blue;">return</span> <span class="o">-</span><span class="nb">abs</span><span class="p">(</span><span class="n">sin</span><span class="p">(</span><span class="n">x0</span><span class="p">)</span><span class="o">*</span><span class="n">cos</span><span class="p">(</span><span class="n">x1</span><span class="p">)</span><span class="o">*</span><span class="n">exp</span><span class="p">(</span><span class="nb">abs</span><span class="p">(</span><span class="mi">1</span><span class="o">-</span><span class="n">sqrt</span><span class="p">(</span><span class="n">x0</span><span class="o">*</span><span class="n">x0</span><span class="o">+</span><span class="n">x1</span><span class="o">*</span><span class="n">x1</span><span class="p">)</span><span class="o">/</span><span class="n">pi</span><span class="p">)))</span>
<span class="n">x</span><span class="p">,</span><span class="n">y</span> <span class="o">=</span> <span class="n">dlib</span><span class="o">.</span><span class="n">find_min_global</span><span class="p">(</span><span class="n">holder_table</span><span class="p">,</span>
<span class="p">[</span><span class="o">-</span><span class="mi">10</span><span class="p">,</span><span class="o">-</span><span class="mi">10</span><span class="p">],</span> <span class="c1" style="color: green;"># Lower bound constraints on x0 and x1 respectively</span>
<span class="p">[</span><span class="mi">10</span><span class="p">,</span><span class="mi">10</span><span class="p">],</span> <span class="c1" style="color: green;"># Upper bound constraints on x0 and x1 respectively</span>
<span class="mi">80</span><span class="p">)</span> <span class="c1" style="color: green;"># The number of times find_min_global() will call holder_table()</span></pre>
<pre style="background-color: white; line-height: 16.25px;"><span class="c1" style="color: green;">
</span></pre>
<pre style="background-color: white; line-height: 16.25px;">Or in C++: </pre>
<pre><span style="color: blue;">auto</span> holder_table <span style="color: #5555ff;">=</span> []<span style="font-family: "lucida console";">(</span><span style="color: blue;"><u>double</u></span> x0, <span style="color: blue;"><u>double</u></span> x1<span style="font-family: "lucida console";">)</span> <b>{</b><span style="color: blue;">return</span> <span style="color: #5555ff;">-</span><span style="color: #bb00bb;">abs</span><span style="font-family: "lucida console";">(</span><span style="color: #bb00bb;">sin</span><span style="font-family: "lucida console";">(</span>x0<span style="font-family: "lucida console";">)</span><span style="color: #5555ff;">*</span><span style="color: #bb00bb;">cos</span><span style="font-family: "lucida console";">(</span>x1<span style="font-family: "lucida console";">)</span><span style="color: #5555ff;">*</span><span style="color: #bb00bb;">exp</span><span style="font-family: "lucida console";">(</span><span style="color: #bb00bb;">abs</span><span style="font-family: "lucida console";">(</span><span style="color: #979000;">1</span><span style="color: #5555ff;">-</span><span style="color: #bb00bb;">sqrt</span><span style="font-family: "lucida console";">(</span>x0<span style="color: #5555ff;">*</span>x0<span style="color: #5555ff;">+</span>x1<span style="color: #5555ff;">*</span>x1<span style="font-family: "lucida console";">)</span><span style="color: #5555ff;">/</span>pi<span style="font-family: "lucida console";">)</span><span style="font-family: "lucida console";">)</span><span style="font-family: "lucida console";">)</span>;<b>}</b>;
<span style="color: #009900;">// obtain result.x and result.y
</span><span style="color: blue;">auto</span> result <span style="color: #5555ff;">=</span> <span style="color: #bb00bb;">find_min_global</span><span style="font-family: "lucida console";">(</span>holder_table,
<b>{</b><span style="color: #5555ff;">-</span><span style="color: #979000;">10</span>,<span style="color: #5555ff;">-</span><span style="color: #979000;">10</span><b>}</b>, <span style="color: #009900;">// lower bounds
</span> <b>{</b><span style="color: #979000;">10</span>,<span style="color: #979000;">10</span><b>}</b>, <span style="color: #009900;">// upper bounds
</span> <span style="color: #bb00bb;">max_function_calls</span><span style="font-family: "lucida console";">(</span><span style="color: #979000;">80</span><span style="font-family: "lucida console";">)</span><span style="font-family: "lucida console";">)</span>;</pre>
<br />
Both of these methods find holder_table's global optima to about 12 digits of precision in about 0.1 seconds. The <a href="http://dlib.net/optimization.html#global_function_search">documentation has much more to say about this new tooling</a>. I'll also make a blog post soon that goes into much more detail on how the method works.<br />
<br />
Finally, here are some fun example outputs from the new semantic segmentation example program:<br />
<img alt="image" src="https://user-images.githubusercontent.com/2297572/32693258-936d34a8-c730-11e7-86fb-b83037fd4bf0.png" style="max-width: 100%;" /><br />
<br />
<img alt="image" src="https://user-images.githubusercontent.com/2297572/32693250-514e6e52-c730-11e7-9187-69f6dc7078f5.png" style="max-width: 100%;" /><br />
<br />
<img alt="image" src="https://user-images.githubusercontent.com/2297572/32693256-8003b61c-c730-11e7-80aa-7cdcd9de4840.png" style="max-width: 100%;" /><br />
<br />
<img alt="image" src="https://user-images.githubusercontent.com/2297572/32693381-e23262d2-c732-11e7-89b4-3b9eacccba4b.png" style="max-width: 100%;" />Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com12tag:blogger.com,1999:blog-6061887630060661987.post-49739638820054161712017-09-23T10:58:00.000-04:002017-09-23T10:58:50.772-04:00Fast Multiclass Object Detection in Dlib 19.7The <a href="http://dlib.net/release_notes.html">new version of dlib</a> is out and the biggest new feature is the ability to train multiclass object detectors with dlib's convolutional neural network tooling. The previous version only allowed you to train single class detectors, but this release adds the option to create single CNN models that output multiple labels. As an example, I created a small 894 image dataset where I annotated the fronts and rears of cars and used it to train a 2-class detector. You can see the resulting detector running in this video:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe width="320" height="266" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/OHbJ7HhbG74/0.jpg" src="https://www.youtube.com/embed/OHbJ7HhbG74?feature=player_embedded" frameborder="0" allowfullscreen></iframe></div>
<br />
If you want to run the car detector from this video on your own images you can check out <a href="http://dlib.net/dnn_mmod_find_cars2_ex.cpp.html">this example program</a>. <br />
<br />
I've also improved the detector speed in dlib 19.7 by pushing more of the processing to the GPU. This makes the detector 2.5x faster. For example, running the detector on the 928x478 image used in <a href="http://dlib.net/dnn_mmod_find_cars_ex.cpp.html">this example program</a> ran at 39fps in the previous version of dlib, but now runs at 98fps (when run on a NVIDIA 1080ti).<br />
<br />
This release also includes a new 5-point face landmarking model that finds the corners of the eyes and bottom of nose:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIWBmhsNWt1CJB6NTzil5ZKJcTWOg-unrU2hMtI2sBs7obsHxeoch23AZvrECX7Vd9om30WewxZjICokGHoDMEpS6NuYPQXcMR_pFPMnZpxoDpycfY84dGz0vgzBv-1zmFMQjGDpbnpok/s1600/therock.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="491" data-original-width="459" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIWBmhsNWt1CJB6NTzil5ZKJcTWOg-unrU2hMtI2sBs7obsHxeoch23AZvrECX7Vd9om30WewxZjICokGHoDMEpS6NuYPQXcMR_pFPMnZpxoDpycfY84dGz0vgzBv-1zmFMQjGDpbnpok/s400/therock.jpg" width="373" /></a></div>
<br />
Unlike the <a href="http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html">68-point landmarking model included with dlib</a>, this model is over 10x smaller at 8.8MB compared to the 68-point model's 96MB. It also runs faster, and even more importantly, works with the <a href="http://blog.dlib.net/2016/10/easily-create-high-quality-object.html">state-of-the-art CNN face detector in dlib</a> as well as the older HOG face detector in dlib. The central use-case of the 5-point model is to perform 2D face alignment for applications like face recognition. In any of the dlib code that does face alignment, the new 5-point model is a drop-in replacement for the 68-point model and in fact is the new recommended model to use with <a href="http://blog.dlib.net/2017/02/high-quality-face-recognition-with-deep.html">dlib's face recognition tooling</a>. <br />
<br />Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com77tag:blogger.com,1999:blog-6061887630060661987.post-7297435057054802122017-08-27T19:47:00.000-04:002017-08-27T19:47:59.555-04:00Vehicle Detection with Dlib 19.5<br />
Dlib v19.5 is out and there are a <a href="http://dlib.net/release_notes.html">lot of new features</a>. There is a dlib to caffe converter, a bunch of new deep learning layer types, cuDNN v6 and v7 support, and a bunch of optimizations that make things run faster in different situations, like ARM NEON support, which makes <a href="http://blog.dlib.net/2014/02/dlib-186-released-make-your-own-object.html">HOG based detectors</a> run a lot faster on mobile devices.<br />
<br />
However, the coolest and most requested feature has been an upgrade to the <a href="http://blog.dlib.net/2016/10/easily-create-high-quality-object.html">CNN+MMOD object detector</a> to support detecting things with varying aspect ratios. The previous version of the detector required the training data to consist of objects that all had essentially the same aspect ratio. This is fine for tasks like face detection and <a href="http://blog.dlib.net/2016/10/hipsterize-your-dog-with-deep-learning.html">dog hipsterization</a>, but obviously not as general as you would like.<br />
<br />
So dlib v19.5 includes an updated version of the MMOD loss layer that can be used to learn an object detector from a dataset with any mixture of bounding box shapes and sizes. To demo this new feature, I used the new MMOD code to create a vehicle detector, which you can see running on these videos. This detector is trained to find cars moving with you in traffic, and therefore cars where the rear end of the vehicle is visible.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/4B3bzmxMAZU/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/4B3bzmxMAZU?feature=player_embedded" width="320"></iframe></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/bP2SUo5vSlc/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/bP2SUo5vSlc?feature=player_embedded" width="320"></iframe></div>
<br />
The detector is just as fast as previous versions of the CNN+MMOD detector. For instance, when I run it on my NVIDIA 1080ti I can process 39 frames per second when processing them individually and 93 frames per second when processing them grouped into batches. This assumes a frame size of 928x478.<br />
<br />
If you want to run this detector yourself you can check out the <a href="http://dlib.net/dnn_mmod_find_cars_ex.cpp.html">new example program</a> that does just that. The detector was trained on a modest dataset of 2217 images, which is also available, as is <a href="http://dlib.net/dnn_mmod_train_find_cars_ex.cpp.html">the training code</a>. Both these new example programs contain a lot of information about training this kind of detector and are worth reading if you want to understand the details involved. However, we can go into a short description here to understand how the detector works.<br />
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCxjb1yUHe8pgTqPeEHaKBpVdjja0BeVD5p1ICegi6-XxRwS-x7i1zOMOnNwwRACnDdLpqpLh-Qx-r29nNLmCnU7RDG-PyI-igR2kRqAoiL10GKxrvfBvpBKP9uEYP5-pwAueVNifuo2Y/s1600/mmod_tutorial_raw_image_with_detections.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="478" data-original-width="928" height="328" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCxjb1yUHe8pgTqPeEHaKBpVdjja0BeVD5p1ICegi6-XxRwS-x7i1zOMOnNwwRACnDdLpqpLh-Qx-r29nNLmCnU7RDG-PyI-igR2kRqAoiL10GKxrvfBvpBKP9uEYP5-pwAueVNifuo2Y/s640/mmod_tutorial_raw_image_with_detections.jpg" width="640" /></a></div>
<br />
Take this image as an example. I ran the new vehicle detector on it and plotted the resulting detections as red boxes. So what are the processing steps that go from the raw image to the 6 boxes? To roughly summarize, they are:<br />
<ol>
<li>Create an image pyramid and pack the pyramid into one big image. Let's call this the "tiled pyramid"</li>
<li>Run the tiled pyramid image through a CNN. The CNN outputs a new image where bright pixels in the output image indicate the presence of cars.</li>
<li>Find pixels in the CNN's output image with a value > 0. Those locations are your preliminary car detections.</li>
<li>Perform non-maximum suppression on the preliminary detections to produce the final output.</li>
</ol>
Steps 3 and 4 are pretty straightforward. It's the first two steps that are complicated. So to understand them, let's visualize the outputs of these first two steps. All step 1 does is call <a href="http://dlib.net/imaging.html#create_tiled_pyramid">dlib::create_tiled_pyramid</a> on the input image to produce this new image:</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj56WQLrGDwXHczaFeVUWMvV6YmRUPuTo_ZVgqTOFdw1h9E024mALSIluFsucttkpId5gniFB0k8xy0aaUnBReLAl1oJQgMJqEf96nZS0AndyqKCGv-yf6zX8eiMTxpwXmXzvIA8kLivzs/s1600/mmod_tutorial_image_pyramid.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="858" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj56WQLrGDwXHczaFeVUWMvV6YmRUPuTo_ZVgqTOFdw1h9E024mALSIluFsucttkpId5gniFB0k8xy0aaUnBReLAl1oJQgMJqEf96nZS0AndyqKCGv-yf6zX8eiMTxpwXmXzvIA8kLivzs/s640/mmod_tutorial_image_pyramid.jpg" width="342" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
What's special about this image is that we don't need to worry about scale anymore. That is, suppose we have a detection algorithm that can find cars, but it only knows how to find cars of a certain size. No problem. When you run it on this tiled pyramid image you are going to find each car somewhere in it at the scale your detector expects. Moreover, the tiled pyramid is only about 3.7 times larger than the original image, so processing it instead of the raw image gives us full scale invariance for only a 3.7x increase in computational cost. That's a very reasonable trade. Moreover, tiling it inside a rectangular image makes it very easy to process using normal CNN tooling on a GPU and still get full GPU speeds. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Now for step 2. The CNN takes the tiled pyramid as input, does a bunch of convolutions, and outputs a new set of images. In the case of our vehicle detector, it outputs 3 new images, each is a detection strength map that gets "hot" in locations likely to contain a vehicle. The reason there are 3 images for the vehicle detector is because there are, roughly, 3 different aspect ratios (tall and skinny e.g. semi trucks, short and wide e.g. sedans, and squarish e.g. SUVs). For purposes of display, I have combined the 3 images into one by taking the pointwise max of the 3 original images. You can see this combined image below. The dark blue areas are places the CNN is saying "definitely not a vehicle" and the bright red locations are the positions it thinks contain a vehicle.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtineTs6MQuaZYARaRwP8DETjr5QYvCqao_eq1NvNFMqrxW5FOksu9N2gAIZ7D7a0C0ujz56ymmQ4c-iQpQBLEN9EtEaT9SjDUbrXiNIG3kQ8_CiH9gX0AJiPpoT0rWTG81Sy8yx2bPt4/s1600/mmod_tutorial_output_tensor.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="848" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtineTs6MQuaZYARaRwP8DETjr5QYvCqao_eq1NvNFMqrxW5FOksu9N2gAIZ7D7a0C0ujz56ymmQ4c-iQpQBLEN9EtEaT9SjDUbrXiNIG3kQ8_CiH9gX0AJiPpoT0rWTG81Sy8yx2bPt4/s640/mmod_tutorial_output_tensor.jpg" width="338" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
If we overlay this CNN output on top of the tiled pyramid you can see it's doing the right thing. The cars get bright red dots on them, right in the centers of the cars. Moreover, you can tell that the CNN is only detecting cars at a certain scale. The smaller cars are detected at the top of the pyramid and only as we progress down the pyramid does it begin to detect the larger cars.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCWVJpceLA78_bCYPGvr_8fn5mCw7R8cbkiMWEJ7j2lITiqaKz8OQyMoZ0hePCUkNroevP6Df8u1dqpbguc2LyQ8ZV3ozzXXRBSZmun30XCLB3og3yB0RvnK2SyQ6aC1W6_CBYA3JlC8Q/s1600/mmod_tutorial_image_pyramid_with_saliency.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="858" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCWVJpceLA78_bCYPGvr_8fn5mCw7R8cbkiMWEJ7j2lITiqaKz8OQyMoZ0hePCUkNroevP6Df8u1dqpbguc2LyQ8ZV3ozzXXRBSZmun30XCLB3og3yB0RvnK2SyQ6aC1W6_CBYA3JlC8Q/s640/mmod_tutorial_image_pyramid_with_saliency.jpg" width="342" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
After the CNN output is obtained, all the detection code needs to do is threshold the CNN output, find all the hot spots, apply non-max suppression, and output the boxes corresponding to the identified hot spots. And that's it, that's all the CNN+MMOD detector is doing.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
On the other hand, describing how the CNN is trained is more complicated. The code in dlib uses the usual stochastic gradient descent methods. You can see many of the details if you read the dlib DNN example programs. How deep learning works in general is a big topic, but the thing most interesting here is the MMOD loss layer. For the gory details on that I refer you to the <a href="https://arxiv.org/abs/1502.00046">MMOD paper</a> which explains the loss function. In the paper it is discussed in the context of networks that are linear in their parameters rather than non-linear in their parameters, as is our CNN here. However, for understanding the loss the difference between linear vs. non-linear is a minor detail. In fact, the loss equations are the same for both cases. The only difference is what kind of optimization algorithms are available for each case. In the linear parameter case you can write a fancy numeric solver capable of solving the problem in a few minutes, but with a non-linear parameterization you have to resort to brute force SGD and GPUs running for many hours. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
But at a very high level, it's running the entire detection process over and over during training, counting the number of detection mistakes (false alarms, missed detections, and duplicate detections), and back-propagating that error gradient through the CNN until the CNN stops messing up. Also, since the MMOD loss layer is counting mistakes after non-max suppression is applied, it knows that it needs to get the CNN to avoid producing high outputs in parts of the image that won't be suppressed by non-max suppression. This is why you see the dark blue areas of "definitely not a car" surrounding each of the car detections. The CNN has learned that it needs to be very careful on the border between "it's a car" and "it's not a car" to avoid accidentally detecting the same car multiple times. </div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
This is perhaps easiest to see if we merge the pyramid layers back into the original image. If we make an image where the pixel value is the max over all scales in the pyramid we get this image:</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCVr1f7_p3oX_2NyWBi9d2V9sfbSErVkuOo32b3-hw8Z7zc6h_8SMoRbFFGcTUlF14d4W7y5rlYz8U1hm0s95rOV2NTNEvns_EZU11FWXKjgq3CSYAkzDal6fTK1eEavTwMTgLIwiSTO8/s1600/mmod_tutorial_saliency.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="478" data-original-width="928" height="328" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCVr1f7_p3oX_2NyWBi9d2V9sfbSErVkuOo32b3-hw8Z7zc6h_8SMoRbFFGcTUlF14d4W7y5rlYz8U1hm0s95rOV2NTNEvns_EZU11FWXKjgq3CSYAkzDal6fTK1eEavTwMTgLIwiSTO8/s640/mmod_tutorial_saliency.jpg" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Here you can clearly see the 6 car hotspots and the dark blue areas of "not a car" immediately surrounding them. Finally, overlaying this on the original image gives this wonderful image:</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOPmEK_p3rGZQjNNROFtrGlArNzTsHRchyphenhyphenUu9zeweNEgQaRjB4oxNVtnlOotUvIDfVxOgkiChyphenhyphengBcqhfmTqKKf_oFwkRkEP3_XZA1sJ-hQoAf9uwNBHu7rP2iU61DUMZA5xFWuxcA-XRo/s1600/mmod_tutorial_raw_image_with_saliency.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="478" data-original-width="928" height="328" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOPmEK_p3rGZQjNNROFtrGlArNzTsHRchyphenhyphenUu9zeweNEgQaRjB4oxNVtnlOotUvIDfVxOgkiChyphenhyphengBcqhfmTqKKf_oFwkRkEP3_XZA1sJ-hQoAf9uwNBHu7rP2iU61DUMZA5xFWuxcA-XRo/s640/mmod_tutorial_raw_image_with_saliency.jpg" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div>
<br />
<br /></div>
Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com50tag:blogger.com,1999:blog-6061887630060661987.post-7544539590650980172017-02-12T13:18:00.000-05:002017-03-30T07:39:58.176-04:00High Quality Face Recognition with Deep Metric Learning<div class="separator" style="clear: both; text-align: left;">
Since the last dlib release, I've been working on adding easy to use <a href="https://github.com/davisking/dlib/blob/master/examples/dnn_metric_learning_ex.cpp">deep metric learning tooling</a> to dlib. Deep metric learning is useful for a lot of things, but the most popular application is face recognition. So obviously I had to add <a href="https://github.com/davisking/dlib/blob/master/examples/dnn_face_recognition_ex.cpp">a face recognition example program</a> to dlib. The new example comes with pictures of bald Hollywood action heroes and uses the provided deep metric model to identify how many different people there are and which faces belong to each person. The input images are shown below along with the four automatically identified face clusters:</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhG7UWXvFGBCSMbzbYCZ_57qkiKY31RHprHpFu3ImTX6PPgAqXMiT-JcTidodXR_8Sqm_hSauGTyQFxgS2mSzUOqrhiKXpv2uC0wYX-aaVq7Ic3B-jFfUkFZyRNYL8q7xnQYTyV1F4BKAk/s1600/deep_learning_magic3.svg.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhG7UWXvFGBCSMbzbYCZ_57qkiKY31RHprHpFu3ImTX6PPgAqXMiT-JcTidodXR_8Sqm_hSauGTyQFxgS2mSzUOqrhiKXpv2uC0wYX-aaVq7Ic3B-jFfUkFZyRNYL8q7xnQYTyV1F4BKAk/s1600/deep_learning_magic3.svg.png" /></a></div>
<br />
<br />
Just like all the <a href="https://github.com/davisking/dlib-models">other example dlib models</a>, the pretrained model used by this example program is in the public domain. So you can use it for anything you want. Also, the model has an accuracy of 99.38% on the standard <a href="http://vis-www.cs.umass.edu/lfw/">Labeled Faces in the Wild</a> benchmark. This is comparable to other state-of-the-art models and means that, given two face images, it correctly predicts if the images are of the same person 99.38% of the time.<br />
<br />
For those interested in the model details, this model is a ResNet network with 29 conv layers. It's essentially a version of the ResNet-34 network from the paper Deep Residual Learning for Image Recognition by He, Zhang, Ren, and Sun with a few layers removed and the number of filters per layer reduced by half.<br />
<br />
The network was trained from scratch on a dataset of about 3 million faces. This dataset is derived from a number of datasets. The face scrub dataset[2], the VGG dataset[1], and then a large number of images I personally scraped from the internet. I tried as best I could to clean up the combined dataset by removing labeling errors, which meant filtering out a lot of stuff from VGG. I did this by repeatedly training a face recognition model and then using graph clustering methods and a lot of manual review to clean up the dataset. In the end, about half the images are from VGG and face scrub. Also, the total number of individual identities in the dataset is 7485. I made sure to avoid overlap with identities in LFW so the LFW evaluation would be valid.<br />
<br />
The network training started with randomly initialized weights and used a structured metric loss that tries to project all the identities into non-overlapping balls of radius 0.6. The loss is basically a type of pair-wise hinge loss that runs over all pairs in a mini-batch and includes hard-negative mining at the mini-batch level. The training code is obviously also available, since that sort of thing is basically the point of dlib. You can find all details on training and model specifics by reading <a href="https://github.com/davisking/dlib/blob/master/examples/dnn_face_recognition_ex.cpp">the example program</a> and consulting the referenced parts of dlib. There is also a <a href="https://github.com/davisking/dlib/blob/master/python_examples/face_recognition.py">Python API</a> for accessing the face recognition model.<br />
<br />
<br />
<br />
[1] O. M. Parkhi, A. Vedaldi, A. Zisserman Deep Face Recognition British Machine Vision Conference, 2015.<br />
[2] H.-W. Ng, S. Winkler. A data-driven approach to cleaning large face datasets. Proc. IEEE International Conference on Image Processing (ICIP), Paris, France, Oct. 27-30, 2014Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com466tag:blogger.com,1999:blog-6061887630060661987.post-77139259744195523822016-10-11T06:59:00.000-04:002016-10-11T06:59:10.365-04:00Easily Create High Quality Object Detectors with Deep LearningA <a href="http://blog.dlib.net/2014/02/dlib-186-released-make-your-own-object.html">few years ago</a> I added an implementation of the <a href="https://arxiv.org/abs/1502.00046">max-margin object-detection algorithm</a> (MMOD) to dlib. This tool has since become quite popular as it frees the user from tedious tasks like hard negative mining. You simply label things in images and it learns to detect them. It also produces high quality detectors from relatively small amounts of training data. For instance, <a href="http://dlib.net/fhog_object_detector_ex.cpp.html">one of dlib's example programs</a> shows MMOD learning a serviceable face detector from only 4 images. <br />
<br />
However, the MMOD implementation in dlib used HOG feature extraction followed by a single linear filter. This means it's incapable of learning to detect objects that exhibit complex pose variation or have a lot of other variability in how they appear. To get around this, users typically train multiple detectors, one for each pose. That works OK in many cases but isn't a really good general solution. Fortunately, over the last few years convolutional neural networks have proven themselves to be capable of dealing with all these issues within a single model. <br />
<br />
So the obvious thing to do was to add an implementation of MMOD with the HOG feature extraction replaced with a convolutional neural network. <a href="http://dlib.net/release_notes.html">The new version of dlib</a>, v19.2, contains just such a thing. <a href="http://dlib.net/dnn_mmod_ex.cpp.html">On this page</a> you can see a short tutorial showing how to train a convolutional neural network using the MMOD loss function. It uses <a href="http://blog.dlib.net/2016/06/a-clean-c11-deep-learning-api.html">dlib's new deep learning API</a> to train the detector end-to-end on the very same 4 image dataset used in the HOG version of the example program. Happily, and very much to the surprise of myself and my colleagues, it learns a working face detector from this tiny dataset. Here is the detector run over an image not in the training data:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMqBOqDZvp8WoU-ueytgjmdIIwxuvUpRjRzfBN9YOHFy0QZb77OEohKNTSUtowaGVckBY6PvS73Bo6NP4Fx0H2Ybzqq-wqMsbxn9o9EvU-3nSwFDW9bniK3m2gcSTnfhnPPHXJjQdvymg/s1600/mmod_example_test_image.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="419" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMqBOqDZvp8WoU-ueytgjmdIIwxuvUpRjRzfBN9YOHFy0QZb77OEohKNTSUtowaGVckBY6PvS73Bo6NP4Fx0H2Ybzqq-wqMsbxn9o9EvU-3nSwFDW9bniK3m2gcSTnfhnPPHXJjQdvymg/s640/mmod_example_test_image.jpg" width="640" /></a></div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVGAAtHvLBnAllH6evfA_iQoB3oovyzNUGbfG9bWM-0iAdOPh-k984R9eEd6TeFC3edAxNEavZ-vLCujn05PrN4-0pu8gY0EJi1mmUFYBYdSCzGEdi6aIMbLlZGBcFLw310mYjyva529w/s1600/MMOD_vs_fasterRCNN_on_FDDB_smaller_image.png" imageanchor="1"></a><br />
I expected the CNN version of MMOD to inherit the low training data requirements of the HOG version of MMOD, but working with only 4 training images is very surprising considering other deep learning methods typically require many thousands of images to produce any kind of sensible results.<br />
<br />
The detector is also reasonably fast for a CNN. On the CPU, it takes about 370ms to process a 640x480 image. On my NVIDIA Titan X GPU (the Maxwell version, not the newer Pascal version) it takes 45ms to process an image when images are processed one at a time. If I group the images into batches then it takes about 18ms per image. <br />
<br />
To really test the new CNN version of MMOD, I ran it through the leading face detection benchmark, <a href="http://vis-www.cs.umass.edu/fddb/">FDDB</a>. This benchmark has two modes, 10-fold cross-validation and unrestricted. Both test on the same dataset, but in the 10-fold cross-validation mode you are only allowed to train on data in the FDDB dataset. In the unrestricted mode you can train on any data you like so long as it doesn't include images from FDDB. I ran the 10-fold cross-validation version of the FDDB challenge. This means I trained 10 CNN face detectors, each on 9 folds and tested on the held out 10th. I did not perform any hyper parameter tuning. Then I ran the results through the FDDB evaluation software and got this plot:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYF8xY_9h5Yw06EAeMEcJNons7trWSl60hzSVaiKwxylWxTe1fMAJZY7l00bqcgpHq9Dmc11U2nWcDxXYSV5RGcXg23e3I2pqSqPCD8o3u7sbLil-Lnl4mlvlvVoftWzd6x9XHgP-vVdY/s1600/MMOD_vs_fasterRCNN_vs_ViolaJones_on_FDDB.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYF8xY_9h5Yw06EAeMEcJNons7trWSl60hzSVaiKwxylWxTe1fMAJZY7l00bqcgpHq9Dmc11U2nWcDxXYSV5RGcXg23e3I2pqSqPCD8o3u7sbLil-Lnl4mlvlvVoftWzd6x9XHgP-vVdY/s640/MMOD_vs_fasterRCNN_vs_ViolaJones_on_FDDB.png" width="640" /></a></div>
<br />
The X axis is the number of false alarms produced over the entire 2845 image dataset. The Y axis is recall, i.e. the fraction of faces found by the detector. The green curve is the new dlib detector, which in this mode only gets about 4600 faces to train on. The red curve is the old Viola Jones detector which is still popular (although it shouldn't be, obviously). Most interestingly, the blue curve is a state-of-the-art result from the paper <a href="https://arxiv.org/abs/1606.03473">Face Detection with the Faster R-CNN</a>, published only 4 months ago. In that paper, they train their detector on the very large WIDER dataset, which consists of 159,424 faces, and arguably get worse results on FDDB than the dlib detector trained on only 4600 faces.<br />
<br />
As another test, I created the <a href="http://blog.dlib.net/2016/10/hipsterize-your-dog-with-deep-learning.html">dog hipsterizer</a>, which I made a post about a few days ago. The hipsterizer used the exact same code and parameter settings to train a dog head detector. The only difference was the training data consisted in 9240 dog heads instead of human faces. That produced the very high quality models used in the hipsterizer. So now we can automatically create fantastic images such as this one :)<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5JUxhTrbm-p_7j5n6lsXH3I_m7oxVtBmdF5aGemaRN6ihm3wYS6ArKM9GGM6Ri6SwEFCk4y0gd5qy6hZi-IlqwZUO_8sRsrl8LJG9QW4llUp7lnPyLCQ089uqbyhHbtL9uLQSj4ylIts/s1600/barkhaus_hip.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5JUxhTrbm-p_7j5n6lsXH3I_m7oxVtBmdF5aGemaRN6ihm3wYS6ArKM9GGM6Ri6SwEFCk4y0gd5qy6hZi-IlqwZUO_8sRsrl8LJG9QW4llUp7lnPyLCQ089uqbyhHbtL9uLQSj4ylIts/s640/barkhaus_hip.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="http://www.thebarkhaus.com/">Barkhaus dogs</a> looking fancy</td></tr>
</tbody></table>
<br />
As one last test of the new CNN MMOD tool I made a <a href="http://dlib.net/files/data/dlib_face_detection_dataset-2016-09-30.tar.gz">dataset of 6975 faces</a>. This dataset is a collection of face images selected from many publicly available datasets (excluding the FDDB dataset). In particular, there are images from ImageNet, AFLW, Pascal VOC, the VGG dataset, WIDER, and face scrub. Unlike FDDB, this new dataset contains faces in a wide range of poses rather than consisting of mostly front facing shots. To give you an idea of what it looks like, here are all the faces in the dataset tightly cropped and tiled into one big image:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaOR45C5hNNjipYxgUVhNbmtm4EMiZxnoDnnKbCm4kTVVKVkL8iuxIKtTZKiZMa0SfPn20igqnXAGqUrYBkcWln_uM81R4fNQaqVlnQbUoM6H1WvftQdAiI_WSoVRLEvvoDj96QQtXFv4/s1600/dlib_faces.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaOR45C5hNNjipYxgUVhNbmtm4EMiZxnoDnnKbCm4kTVVKVkL8iuxIKtTZKiZMa0SfPn20igqnXAGqUrYBkcWln_uM81R4fNQaqVlnQbUoM6H1WvftQdAiI_WSoVRLEvvoDj96QQtXFv4/s640/dlib_faces.jpg" width="632" /></a></div>
<br />
Using the new dlib tooling I trained a CNN on this dataset using the same exact code and parameter settings as used by the dog hipsterizer and previous FDDB experiment. If you want to run that CNN on your own images you can use <a href="http://dlib.net/dnn_mmod_face_detection_ex.cpp.html">this example program</a>. I tested this CNN on FDDB's unrestricted protocol and found that it has a recall of 0.879134, which is quite good. However, it produced 90 false alarms. Which sounds bad, until you look at them and find that it's finding labeling errors in FDDB. The following image shows all the "false alarms" it outputs on FDDB. All but one of them are actually faces.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvbwXjUH8_Q_TUAExUrNhGU0KtdgZSfIq2TEelUaLQMnRJFWFhbAGoOgU-GJTHAwJhmYqs3IIvTAYgpIrlF4h9kVmvM3ij7J-gmcagD4P1WFtNuRjbR3cqNWgvqA6ohDB1XBWjWM2xaws/s1600/mmod_cnn_false_alarms_on_FDDB.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="568" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvbwXjUH8_Q_TUAExUrNhGU0KtdgZSfIq2TEelUaLQMnRJFWFhbAGoOgU-GJTHAwJhmYqs3IIvTAYgpIrlF4h9kVmvM3ij7J-gmcagD4P1WFtNuRjbR3cqNWgvqA6ohDB1XBWjWM2xaws/s640/mmod_cnn_false_alarms_on_FDDB.jpg" width="640" /></a></div>
<br />
Finally, to give you a more visceral idea of the difference in capability between the new CNN detector and the old HOG detector, here are a few images where I ran dlib's default HOG face detector (which is actually 5 HOG models) and the <a href="http://dlib.net/dnn_mmod_face_detection_ex.cpp.html">new CNN face detector</a>. The red boxes are CNN detections and blue boxes are from the older HOG detector. While the HOG detector does an excellent job on easy faces looking at the camera, you can see that the CNN is way better at handling not just the easy cases but all faces in general. And yes, I ran the HOG detector on all the images, it's just that it fails to find any faces in some of them.<br />
<div>
<br /></div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTWKufyktuc6D80ts5kaFVycMxSrPfkymFnpYe3L_jXXrCZKHhxE3jRelMs-EtbHvR7TBdmuP9Lcu5A4pd44u1pSHcXN0wjpsoykteX7YfxwZyBw_EXgpICOSU_wf6uIkNmlpV4MiitUg/s1600/Screenshot+from+2016-10-02+09%253A43%253A22.png" imageanchor="1" style="clear: right; display: inline !important; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="249" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTWKufyktuc6D80ts5kaFVycMxSrPfkymFnpYe3L_jXXrCZKHhxE3jRelMs-EtbHvR7TBdmuP9Lcu5A4pd44u1pSHcXN0wjpsoykteX7YfxwZyBw_EXgpICOSU_wf6uIkNmlpV4MiitUg/s320/Screenshot+from+2016-10-02+09%253A43%253A22.png" width="320" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWnxyKE66-mZ3v3tieRJuRXvYyQHhANhfwulrARie2GNrSo5OCNr6cSldCEAkafmGPWFuMJEq6eZIDPWYRkwfsgnOl05GhYV9oTU4YwyEIlkE047P2K7jwu8qo5wzD3CNQbSS0_AuV2jk/s1600/Screenshot+from+2016-10-02+09%253A43%253A16.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWnxyKE66-mZ3v3tieRJuRXvYyQHhANhfwulrARie2GNrSo5OCNr6cSldCEAkafmGPWFuMJEq6eZIDPWYRkwfsgnOl05GhYV9oTU4YwyEIlkE047P2K7jwu8qo5wzD3CNQbSS0_AuV2jk/s320/Screenshot+from+2016-10-02+09%253A43%253A16.png" width="320" /></a><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjJOa55DSLDtUOpHmT4TqzBL_6yoPKW7IyvshxVUlyyapxZEVNyyVBKK06KgLhKCjrn0lZ5aCG1FdXJcQEd1NR7-8oczCwqDeBCNm0G61Sgsnj0dkWIknqV4v1mYyJsbalXRMokmsxNyI/s1600/Screenshot+from+2016-10-02+09%253A43%253A28.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="224" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjJOa55DSLDtUOpHmT4TqzBL_6yoPKW7IyvshxVUlyyapxZEVNyyVBKK06KgLhKCjrn0lZ5aCG1FdXJcQEd1NR7-8oczCwqDeBCNm0G61Sgsnj0dkWIknqV4v1mYyJsbalXRMokmsxNyI/s320/Screenshot+from+2016-10-02+09%253A43%253A28.png" width="320" /></a><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_NzdmtUCndnQ4c8j9iiCiBMktdqsE1YanC1QG5OnC2iDNq3NFNI2FfoM3DvUsiwzK7nW6Qi1GDDlDr6fMuUccsunGm1n-Cb7hgQOHtTkrNrqeun1OCx6vg2jYxrWu5u7QlmCTRtVok6g/s1600/Screenshot+from+2016-10-02+09%253A43%253A33.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="241" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_NzdmtUCndnQ4c8j9iiCiBMktdqsE1YanC1QG5OnC2iDNq3NFNI2FfoM3DvUsiwzK7nW6Qi1GDDlDr6fMuUccsunGm1n-Cb7hgQOHtTkrNrqeun1OCx6vg2jYxrWu5u7QlmCTRtVok6g/s320/Screenshot+from+2016-10-02+09%253A43%253A33.png" width="320" /></a><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhc3ZBAm3M3t4vc_z8xXedCaqTBwxO_acrE4FiAhMqTCnltIRVfRxzyLvBOPnrPHzN0eN0TLAXEMyyXyTYUwKcMdJWuTv9nW9IOYckgZ-wym24IrWt9qyMgolZiytx3OipexO_QKBaZJOE/s1600/Screenshot+from+2016-10-10+17%253A55%253A51.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhc3ZBAm3M3t4vc_z8xXedCaqTBwxO_acrE4FiAhMqTCnltIRVfRxzyLvBOPnrPHzN0eN0TLAXEMyyXyTYUwKcMdJWuTv9nW9IOYckgZ-wym24IrWt9qyMgolZiytx3OipexO_QKBaZJOE/s320/Screenshot+from+2016-10-10+17%253A55%253A51.png" width="320" /></a></div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifTSgKEpdf5CtE10Q9pp0bYRQiXKAcQZHq7Q6RsQbOInzAkEl6eqY8svH3yZaWaT_zGjSnIG8puXTvJv4C_xv0r3cBDlA50fFhMFtSvG0hWLfTuL9IlVGNq9PmLey8nnBMl8cDVmPEO8Q/s1600/Screenshot+from+2016-10-02+09%253A43%253A40.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="212" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifTSgKEpdf5CtE10Q9pp0bYRQiXKAcQZHq7Q6RsQbOInzAkEl6eqY8svH3yZaWaT_zGjSnIG8puXTvJv4C_xv0r3cBDlA50fFhMFtSvG0hWLfTuL9IlVGNq9PmLey8nnBMl8cDVmPEO8Q/s320/Screenshot+from+2016-10-02+09%253A43%253A40.png" width="320" /></a>Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com254tag:blogger.com,1999:blog-6061887630060661987.post-85092188419489995272016-10-07T21:24:00.000-04:002020-04-17T06:04:15.610-04:00Hipsterize Your Dog With Deep LearningI'm getting ready to make the next <a href="http://dlib.net/">dlib</a> release, which should be out in a few days, and I thought I would point out a humorous new example program. The <a href="https://github.com/davisking/dlib/blob/master/examples/dnn_mmod_dog_hipsterizer.cpp">dog hipsterizer</a>!<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmP2GfHn6BUqLE0k3BFGWPiwzKUxrIXXnXUFkGNtISh5dzBfgEFaDPSDFw8doqks308_koNWi0PGGnrAuXu0vu-WZ31qPSvxTlok9SqbJ_C514MwBQ5S1Q2esLlF9EbyDYhxOhGwv6dLs/s1600/dog_hipsterizer.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="176" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmP2GfHn6BUqLE0k3BFGWPiwzKUxrIXXnXUFkGNtISh5dzBfgEFaDPSDFw8doqks308_koNWi0PGGnrAuXu0vu-WZ31qPSvxTlok9SqbJ_C514MwBQ5S1Q2esLlF9EbyDYhxOhGwv6dLs/s640/dog_hipsterizer.png" width="640" /></a></div>
<br />
It uses dlib's new <a href="http://blog.dlib.net/2016/06/a-clean-c11-deep-learning-api.html">deep learning tools</a> to detect dogs looking at the camera. Then it uses the <a href="http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html">dlib shape predictor</a> to identify the positions of the eyes, nose, and top of the head. From there it's trivial to make your dog hip with glasses and a mustache :)<br />
<br />
This is what you get when you run the <a href="https://github.com/davisking/dlib/blob/master/examples/dnn_mmod_dog_hipsterizer.cpp">dog hipsterizer</a> on this awesome image:<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOG-SGbfMy4OuVOeaDGuU-zWoV-S6ZiZ_R4_9ZjFEtrZi_QR1xrV2sF6IOi1KPZk6HFn6_keCiO9wTVCY1NTlqSk1Vugho4XeLAoXZ3jckQSsCJV8h7i__rnx77kErPXschyphenhyphenoh-Fbd1Io/s1600/barkhaus_hip.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOG-SGbfMy4OuVOeaDGuU-zWoV-S6ZiZ_R4_9ZjFEtrZi_QR1xrV2sF6IOi1KPZk6HFn6_keCiO9wTVCY1NTlqSk1Vugho4XeLAoXZ3jckQSsCJV8h7i__rnx77kErPXschyphenhyphenoh-Fbd1Io/s640/barkhaus_hip.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="http://www.thebarkhaus.com/">Barkhaus dogs</a> looking fancy</td></tr>
</tbody></table>
<br />Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com31tag:blogger.com,1999:blog-6061887630060661987.post-92223819869274347242016-08-13T14:48:00.000-04:002016-08-13T14:52:50.408-04:00Dlib 19.1 ReleasedcuDNN 5.1 is out and it isn't completely backwards compatible with cuDNN 5.0 due to a bug in cuDNN 5.1. For the curious, in cuDNN 5.1 <span style="background-color: white; color: #222222; font-family: "arial" , sans-serif; font-size: 12.8px;">cudnnGetConvolutionBackwardFil</span><wbr style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;"></wbr><span style="background-color: white; color: #222222; font-family: "arial" , sans-serif; font-size: 12.8px;">terAlgorithm() will select the winograd algorithm even when the conv descriptor has a stride not equal to 1, which is an error according to the cuDNN documentation. If you then try to run the winograd algorithm, which is what </span><span style="background-color: white; color: #222222; font-family: "arial" , sans-serif; font-size: 12.8px;">cudnnGetConvolutionBackwardFil</span><wbr style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;"></wbr><span style="background-color: white; color: #222222; font-family: "arial" , sans-serif; font-size: 12.8px;">terAlgorithm() says to do, it leads to the wrong outputs and things don't work. Fortunately, this was detected by dlib's unit tests :)</span><br />
<br />
Therefore, dlib has been updated to work with cuDNN 5.1 and hence we have a dlib 19.1 release, which you can download from <a href="http://dlib.net/">dlib's home page</a>.<br />
<br />
I also recently realized that the fancy std::async() in C++11, an API for launching asynchronous tasks, is not backed by any kind of load balancing at all. For example, if you call std::async() at a faster rate than the tasks complete then your program will create an unbounded number of threads, leading to an eventual crash. That's awful. But std::async() is a nice API and I want to use it. So dlib now contains <a href="http://dlib.net/dlib/threads/async_abstract.h.html">dlib::async()</a> which has the same interface, except instead of the <a href="http://eli.thegreenplace.net/2016/the-promises-and-challenges-of-stdasync-task-based-parallelism-in-c11/">half baked</a> launch policy as the first argument, dlib::async() takes a dlib::thread_pool, giving dlib::async() all the bounded resource use properties of dlib::thread_pool. Moreover, if you don't give dlib::async() a thread pool it will default to a global thread pool instance that contains std::thread::hardware_concurrency() threads. Yay.Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com24tag:blogger.com,1999:blog-6061887630060661987.post-69010443876411396762016-06-26T08:29:00.002-04:002016-06-27T06:48:19.739-04:00A Clean C++11 Deep Learning API<a href="http://dlib.net/">Dlib</a> 19.0 is out and it has a lot of new features, like new elastic net and quadratic program solvers. But the feature I'm most excited about is the new deep learning API. There are a lot of existing deep learning frameworks, but none of them have clean C++ APIs. You have to use them through a language like Python or Lua, which is fine in and of itself. But if you are a professional software engineer working on embedded computer vision projects you are probably working in C++, and using those tools in these kinds of applications can be frustrating. <br />
<br />
So if you use C++ to do computer vision work then dlib's deep learning framework is for you. It makes heavy use of C++11 features, allowing it to expose a very clean and lightweight API. For example, the venerable LeNet can be defined in pure C++ with a using statement:<br />
<div>
<div>
<br /></div>
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuXA7qRCTB8D_RFO3ISRFI3rv_FTgFpoJtdmL1cfsQ1kuuBx8cG_gTvsorcSbyKVd-7vVE0Hew5T02lpcXRWFamILH5Kjz7afNOmL9szas2W6waCJQute7oS_P0LyBYr92RRrNaeUnPps/s1600/lenet5.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuXA7qRCTB8D_RFO3ISRFI3rv_FTgFpoJtdmL1cfsQ1kuuBx8cG_gTvsorcSbyKVd-7vVE0Hew5T02lpcXRWFamILH5Kjz7afNOmL9szas2W6waCJQute7oS_P0LyBYr92RRrNaeUnPps/s1600/lenet5.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: small;">LeNet</span></td></tr>
</tbody></table>
<div>
<br class="Apple-interchange-newline" /></div>
<div>
<pre> <span style="color: blue;">using</span> LeNet <span style="color: #5555ff;">=</span> loss_multiclass_log<span style="color: #5555ff;"><</span>
fc<span style="color: #5555ff;"><</span><span style="color: #979000;">10</span>,
relu<span style="color: #5555ff;"><</span>fc<span style="color: #5555ff;"><</span><span style="color: #979000;">84</span>,
relu<span style="color: #5555ff;"><</span>fc<span style="color: #5555ff;"><</span><span style="color: #979000;">120</span>,
max_pool<span style="color: #5555ff;"><</span><span style="color: #979000;">2</span>,<span style="color: #979000;">2</span>,<span style="color: #979000;">2</span>,<span style="color: #979000;">2</span>,relu<span style="color: #5555ff;"><</span>con<span style="color: #5555ff;"><</span><span style="color: #979000;">16</span>,<span style="color: #979000;">5</span>,<span style="color: #979000;">5</span>,<span style="color: #979000;">1</span>,<span style="color: #979000;">1</span>,
max_pool<span style="color: #5555ff;"><</span><span style="color: #979000;">2</span>,<span style="color: #979000;">2</span>,<span style="color: #979000;">2</span>,<span style="color: #979000;">2</span>,relu<span style="color: #5555ff;"><</span>con<span style="color: #5555ff;"><</span><span style="color: #979000;">6</span>,<span style="color: #979000;">5</span>,<span style="color: #979000;">5</span>,<span style="color: #979000;">1</span>,<span style="color: #979000;">1</span>,
input<span style="color: #5555ff;"><</span>matrix<span style="color: #5555ff;"><</span><span style="color: blue;"><u>unsigned</u></span> <span style="color: blue;"><u>char</u></span><span style="color: #5555ff;">></span><span style="color: #5555ff;">></span><span style="color: #5555ff;">></span><span style="color: #5555ff;">></span><span style="color: #5555ff;">></span><span style="color: #5555ff;">></span><span style="color: #5555ff;">></span><span style="color: #5555ff;">></span><span style="color: #5555ff;">></span><span style="color: #5555ff;">></span><span style="color: #5555ff;">></span><span style="color: #5555ff;">></span><span style="color: #5555ff;">></span><span style="color: #5555ff;">></span>;</pre>
</div>
<br />
Then, using it to train and test a neural network looks like this:<br />
<pre></pre>
<pre> LeNet net;
dnn_trainer<span style="color: #5555ff;"><</span>LeNet<span style="color: #5555ff;">></span> <span style="color: #bb00bb;">trainer</span><span style="font-family: "lucida console";">(</span>net<span style="font-family: "lucida console";">)</span>;
trainer.<span style="color: #bb00bb;">set_learning_rate</span><span style="font-family: "lucida console";">(</span><span style="color: #979000;">0.01</span><span style="font-family: "lucida console";">)</span>;
trainer.<span style="color: #bb00bb;">set_min_learning_rate</span><span style="font-family: "lucida console";">(</span><span style="color: #979000;">0.00001</span><span style="font-family: "lucida console";">)</span>;
trainer.<span style="color: #bb00bb;">set_mini_batch_size</span><span style="font-family: "lucida console";">(</span><span style="color: #979000;">128</span><span style="font-family: "lucida console";">)</span>;
trainer.<span style="color: #bb00bb;">train</span><span style="font-family: "lucida console";">(</span>training_images, training_labels<span style="font-family: "lucida console";">)</span>;
<span style="color: #009900;">// Ask the net to predict labels for all the testing images</span>
<span style="color: blue;">auto</span> predicted_labels <span style="color: #5555ff;">=</span> <span style="color: #bb00bb;">net</span><span style="font-family: "lucida console";">(</span>testing_images<span style="font-family: "lucida console";">)</span>;</pre>
<br />
Dlib will even automatically switch to lower learning rates when the training error stops improving, so you won't have to fiddle with learning rate schedules. The API will certainly let you do so if you want that control. But I've been able to train a number of state-of-the-art ImageNet models without any manual fiddling of learning rates, which I find to be very convenient.<br />
<br />
Depending on how you compile dlib, it will use either the CPU or <a href="https://developer.nvidia.com/cudnn">cuDNN</a> v5. It also supports using multiple GPUs during training and has a "fast mode" and a "low VRAM" mode. Compared to <a href="http://caffe.berkeleyvision.org/">Caffe</a>, dlib's fast mode is about 1.6x times faster than Caffe but uses about 1.5x as much VRAM, while the low VRAM mode is about 0.85x the speed of Caffe but uses half the VRAM as Caffe. So dlib's new deep learning API is fast but can also let you run larger models in the same amount of VRAM if you are VRAM constrained.<br />
<br />
It's also fully documented. The basics are covered in <a href="http://dlib.net/dnn_introduction_ex.cpp.html">this tutorial</a> and then more advanced concepts are covered in a <a href="http://dlib.net/dnn_introduction2_ex.cpp.html">follow on tutorial</a>. These tutorials show how to define LeNet and ResNet architectures in dlib and another tutorial shows <a href="http://dlib.net/dnn_inception_ex.cpp.html">how to define Inception networks</a>. And even more importantly, every function and class in the API is documented in the reference material. Moreover, if you want to <a href="http://dlib.net/dlib/dnn/layers_abstract.h.html#EXAMPLE_COMPUTATIONAL_LAYER_">define your own computational layers</a>, <a href="http://dlib.net/dlib/dnn/loss_abstract.h.html#EXAMPLE_LOSS_LAYER_">loss layers</a>, <a href="http://dlib.net/dlib/dnn/input_abstract.h.html#EXAMPLE_INPUT_LAYER">input layers</a>, or <a href="http://dlib.net/dlib/dnn/solvers_abstract.h.html#EXAMPLE_SOLVER">solvers</a>, you can because the interfaces you have to implement are fully documented.<br />
<br />
I've also included a pretrained ResNet34A model and <a href="http://dlib.net/dnn_imagenet_ex.cpp.html">this example</a> shows how to use it to classify images. This pretrained model has a top5 error of 7.572% on the 2012 imagenet validation dataset, which is slightly better than the results reported in the original paper Deep Residual Learning for Image Recognition by He, Zhang, Ren, and Sun. Training this model took about two weeks while running on a single Titan X GPU. <br />
<br />
To use the new deep learning tools, all you need to install is cuDNN v5. Then you can compile the dlib example programs using the <a href="http://dlib.net/compile.html">normal CMake commands</a>. There are no other dependencies. In fact, if you don't install cuDNN CMake will automatically configure dlib to use only the CPU and the examples will still run (but much slower). You will however need a C++11 compiler, which precludes current versions of visual studio since they shamefully <i>still</i> lack full C++11 support. But any mildly recent version of GCC will work. Also, you can use visual studio with the non-DNN parts of dlib as they don't require C++11 support.<br />
<br /></div>
<div>
Finally, development of this new deep learning toolkit was sponsored by <a href="http://www.stresearch.com/">Systems & Technology Research</a>, as part of the <a href="https://www.iarpa.gov/index.php/research-programs/janus">IARPA JANUS project</a>. Without their support and feedback it wouldn't be nearly as polished and flexible. <a href="http://www.jeffreybyrne.com/">Jeffrey Byrne</a> in particular was instrumental in finding bugs and usability problems in early versions of the API.<br />
<br />
<br /></div>
<div>
<br />
<br />
<br /></div>
</div>
Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com26tag:blogger.com,1999:blog-6061887630060661987.post-27297852719070711272015-06-05T18:45:00.001-04:002015-06-06T09:30:03.074-04:00Reinforcement Learning, Control, and 3D VisualizationOver the last few months I've spent a lot of time studying optimal control and reinforcement learning. Aside from reading, one of the best ways to learn about something is to do it yourself, which in this case means a lot of playing around with the well known algorithms, and for those I really like, including them into <a href="http://dlib.net/">dlib</a>, which is the subject of this post. So far I've added two methods, the first, added in a previous dlib release was the well known <a href="http://dlib.net/ml.html#lspi">least squares policy iteration</a> reinforcement learning algorithm. The second, and my favorite so far due to its practicality, is a tool for solving <a href="http://en.wikipedia.org/wiki/Model_predictive_control">model predictive control</a> problems.<br />
<br />
There is a dlib <a href="http://dlib.net/mpc_ex.cpp.html">example program</a> that explains the new model predictive control tool in detail. But the basic idea is that it takes as input a simple linear equation defining how some process evolves in time and then tells you what control input you should apply to make the process go into some user specified state. For example, imagine you have an air vehicle with a rocket on it and you want it to hover at some specific location in the air. You could use a model predictive controller to find out what direction to fire the rocket at each moment to get the desired outcome. In fact, the dlib example program is just that. It produces the following visualization where the vehicle is the black dot and you want it to hover at the green location. The rocket thrust is shown as the red line:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5KpCkAYQ6nAZynRhyxy5Vf-vrSu-VK36WXktlEErStRChkrhV3xrMjEITuCGWHlf5DEhEHjRakcAty-KsUccKYjsOOHpiGnkVS0vHMm5OdwXDZtUVvXnUeya8F0wN4hdkd_KhUsQkF2g/s1600/mpc.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5KpCkAYQ6nAZynRhyxy5Vf-vrSu-VK36WXktlEErStRChkrhV3xrMjEITuCGWHlf5DEhEHjRakcAty-KsUccKYjsOOHpiGnkVS0vHMm5OdwXDZtUVvXnUeya8F0wN4hdkd_KhUsQkF2g/s1600/mpc.gif" /></a></div>
<br />
<br />
Another fun new tool in dlib is the <a href="http://dlib.net/3d_point_cloud_ex.cpp.html">perspective_window</a>. It's a super easy to use tool for visualizing 3D point cloud data. For instance, the included example program shows how to make this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjt4u5_uUcjqMb3cMyNmN7jlE6KNVtn79d01OAbg-W8nqTuZBp01rL7IZnr7WTCrYSfFgfDK08BLSqi8bFt_BEclRoR45aA-0WHOOHJOlt_bQwP-QEsr38brqX8acKWYuRuYAzcpKF7WnE/s1600/perspective_window.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="267" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjt4u5_uUcjqMb3cMyNmN7jlE6KNVtn79d01OAbg-W8nqTuZBp01rL7IZnr7WTCrYSfFgfDK08BLSqi8bFt_BEclRoR45aA-0WHOOHJOlt_bQwP-QEsr38brqX8acKWYuRuYAzcpKF7WnE/s320/perspective_window.gif" width="320" /></a></div>
<br />
Finally, Patrick Snape contributed Python bindings for <a href="http://blog.dlib.net/2015/02/dlib-1813-released.html">dlib's video tracker</a>, so now you can <a href="http://dlib.net/correlation_tracker.py.html">use it from Python</a>. To try out these new tools download the <a href="https://github.com/davisking/dlib/releases/download/v18.16/dlib-18.16.tar.bz2">newest dlib release</a>.<br />
<br />
<br />
<br />Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com9tag:blogger.com,1999:blog-6061887630060661987.post-85231594533106838412015-02-03T20:08:00.004-05:002015-03-02T06:53:16.632-05:00Python Stuff and Real-Time Video Object TrackingThe new version of <a href="http://dlib.net/">dlib</a> is out today. As promised, there is now a full Python API for using dlib's <a href="http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html">state-of-the-art object pose estimation and learning</a> tools. You can see examples of this API <a href="http://dlib.net/face_landmark_detection.py.html">here</a> and <a href="http://dlib.net/train_shape_predictor.py.html">here</a>. Thank Patrick Snape, one of the main developers of the <a href="http://www.menpo.org/">menpo</a> project, for this addition.<br />
<br />
Also, I've added an implementation of the winning algorithm from last year's <a href="http://www.votchallenge.net/vot2014/">Visual Object Tracking Challenge</a>. This was a method described in the paper:<br />
<blockquote class="tr_bq">
<span style="background-color: white;">Danelljan, Martin, et al. "Accurate scale estimation for robust visual tracking." Proceedings of the British Machine Vision Conference BMVC. 2014.</span></blockquote>
You can see some videos showing dlib's implementation of this new tracker in action on youtube:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/-8-KCoOFfqs/0.jpg" frameborder="0" height="266" src="http://www.youtube.com/embed/-8-KCoOFfqs?feature=player_embedded" width="320"></iframe></div>
<br />
All these videos were processed by exactly the same piece of software. No hand tweaking or any funny business. The only required input (other than the raw video) is a bounding box on the first frame and then the tracker automatically follows whatever is inside the box after that. The whole thing runs at over 150fps on my desktop. You can see an example program showing how to use it <a href="http://dlib.net/video_tracking_ex.cpp.html">here</a>, or just go <a href="http://sourceforge.net/projects/dclib/files/latest/download">download the new dlib instead</a> :)<br />
<br />
I've also finally posted the paper I've been writing on dlib's <a href="http://arxiv.org/abs/1502.00046">structural SVM based training algorithm</a>, which is the algorithm behind the <a href="http://blog.dlib.net/2014/02/dlib-186-released-make-your-own-object.html">easy to use object detector</a>.Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com88tag:blogger.com,1999:blog-6061887630060661987.post-66129904732117853432014-12-20T16:34:00.001-05:002014-12-20T16:34:39.737-05:00Dlib 18.12 released<span style="background-color: white; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; line-height: 18.4799995422363px;">I just released the next version of <a href="http://dlib.net/">dlib</a>. This time I added </span><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif;"><span style="line-height: 18.4799995422363px;">tools for computing 2D FFTs, Hough transforms, image </span></span><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; line-height: 18.4799995422363px;">skeletonizations, and also a simple and type safe API for calling C++ code from </span><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; line-height: 18.4799995422363px;">MATLAB. Readers familiar with writing MATLAB mex functions know how much of a pain it is, but no longer! Here is an example of a C++ function callable from MATLAB using <a href="http://dlib.net/dlib/matlab/example_mex_function.cpp.html">dlib's new MATLAB binding API</a>. You can also compile it with CMake so building it is super easy. There is an example CMake file in the dlib/matlab folder showing how to set it up. I also used this tool to give the <a href="http://blog.dlib.net/2014/04/mitie-completely-free-and-state-of-art.html">MITIE project</a> a simple MATLAB API. So you can see another example of how easy it is to set this up in the <a href="https://github.com/mit-nlp/MITIE/tree/master/examples/matlab">MITIE MATLAB example</a></span><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif;"><span style="line-height: 18.4799995422363px;">. </span></span><br />
<span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif;"><span style="line-height: 18.4799995422363px;"><br /></span></span>
<span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif;"><span style="line-height: 18.4799995422363px;">There are also some fun new things in the pipe for the next dlib release (v18.13). First, Patrick Snape, one of the main developers of the <a href="http://www.menpo.io/">menpo</a> project, is adding a Python interface to dlib's <a href="http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html">shape prediction</a> tools. You can follow that over on <a href="https://github.com/davisking/dlib/pull/2">dlib's github repo</a>. I'm also working on a single object tracker for <a href="http://code.opencv.org/projects/opencv/wiki/VisionChallenge">OpenCV's Vision Challenge</a> which I plan to include in the next version of dlib.</span></span>Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com17tag:blogger.com,1999:blog-6061887630060661987.post-34319112543811145422014-11-15T10:32:00.001-05:002014-11-16T08:38:23.662-05:00Dlib 18.11 releasedThe new version of <a href="http://dlib.net/">dlib</a> is out. This release contains mostly minor bug fixes and usability improvements, with the notable exception of new routines for extracting local-binary-pattern features from images and improved tools for learning distance metrics. See the <a href="http://dlib.net/release_notes.html">release notes</a> for further information.<br />
<br />
I also recently found out about two particularly interesting projects that use dlib. The first is <a href="http://www.menpo.io/">menpo</a>, a Python library focused on computer vision which is being developed by a team at Imperial College London. If you are interested in a Python library that pulls together a bunch of computer vision tools then definitely check it out. The other interesting project is <a href="http://www.ceemple.com/">Ceemple</a>, which is basically an interactive language shell for C++. They have integrated a bunch of libraries like dlib and OpenCV into it with the general goal of making C++ development feel more rapid and interactive. So think of something like MATLAB or IPython, but for C++.Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com1tag:blogger.com,1999:blog-6061887630060661987.post-91076261705721029102014-10-21T10:41:00.001-04:002014-10-24T06:17:04.399-04:00MITIE v0.3 Released: Now with Java and R APIs<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="background-color: white; color: #222222; line-height: 18.4799995422363px;">We just made the next release of MITIE, a new </span><a href="https://github.com/mit-nlp/MITIE" style="background-color: white; color: #667799; line-height: 18.4799995422363px; text-decoration: none;">DARPA funded information extraction tool</a><span style="background-color: white; color: #222222; line-height: 18.4799995422363px;"> being created by our team at MIT. This release is relatively minor and just adds APIs for Java and R. The project page on </span><a href="https://github.com/mit-nlp/MITIE">github</a> explains how to get started using either of these APIs. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">I want to take some time and explain how the Java API is implemented since, as I discovered while making MITIE's Java API, there aren't clear instructions for doing this anywhere on the internet. So hopefully this little tutorial will help you if you decide to make a similar Java binding to a C++ library. So to begin, let's think about the requirements for a good Java binding:</span><br />
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">You should be able to compile it from source with a simple command</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">A user of your library should not need to edit or configure anything to compile the API</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The compilation process should work on any platform</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Writing JNI is awful so you shouldn't have to do that</span></li>
</ul>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This pretty much leads you to <a href="http://www.swig.org/">Swig </a>and <a href="http://www.cmake.org/">CMake</a> which are both great tools. However, finding out how to get CMake to work with Swig was painful and is pretty much what this blog post is about. Happily, it's possible to do and results in a very clean and easy to use mechanism for creating Java APIs. In particular, you can compile <a href="https://github.com/mit-nlp/MITIE/tree/master/mitielib/java">MITIE's Swig/CMake based Java API</a> using the usual CMake commands:</span><br />
<pre class="code_box">mkdir build
cd build
cmake ..
cmake --build . --config Release --target install</pre>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">That creates a jar file and shared library file which together form the MITIE Java API. Let's run through a little example to see how you can define new Java APIs. Imagine you have created a simple C++ API that looks like this:</span></div>
<div>
<pre class="code_box">void printSomeString (const std::string& message);
class MyClass {
public:
std::vector<std::string> getSomeStrings() const;
};</pre>
</div>
<div>
</div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">and you want to be able to use it from Java. You just need to put this C++ API in a header file called swig_api.h and include some Swig commands that tell it what to call std::vector<std::string> in the generated Java API. So the contents of swig_api.h would look like:</span></div>
<div>
<pre class="code_box">// Define some swig type maps that tell swig what to call various instantiations of
// std::vector.
#ifdef SWIG
%include "std_string.i"
%include "std_vector.i"
%template(StringVector) std::vector<std::string>;
#endif
#include <string>
#include <vector>
void printSomeString (const std::string& message);
class MyClass {
public:
std::vector<std::string> getSomeStrings() const;
};
</pre>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">The next step is to create a CMakeLists.txt file that tells CMake how to compile your API. In our case, it would look like:</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">
</span>
<br />
<pre class="code_box">cmake_minimum_required (VERSION 2.8.4)
project(example)
set(java_package_name edu.mit.ll.example)
# List the source files you want to compile into the Java API. These contain
# things like implementations of printSomeString() and whatever else you need.
set(source_files my_source.cpp another_source_file.cpp )
# List the folders that contain your header files
include_directories( . )
# List of libraries to link to. For example, you might need to link to pthread
set(additional_link_libraries pthread)
# Tell CMake to put the compiled shared library and example.jar file into the
# same folder as this CMakeLists.txt file when the --target install option is
# executed. You can put any folder here, just give a path that is relative to
# the CMakeLists.txt file.
set(install_target_output_folder .)
include(cmake_swig_jni)</pre>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">That's it. Now you can compile your Java API using CMake and you will get an example.jar and example.dll or libexample.so file depending on your platform. Then to use it you can write java code like this:</span></div>
</div>
<div>
<pre class="code_box">import edu.mit.ll.example.*;
public class Example {
public static void main(String args[]) {
global.printSomeString("hello world!");
MyClass obj = new MyClass();
StringVector temp = obj.getSomeStrings();
for (int i = 0; i < temp.size(); ++i)
System.out.println(temp.get(i));
}
}</pre>
</div>
<div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">and execute it via:</span></div>
<pre class="code_box"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">javac -classpath example.jar Example.java
java -classpath example.jar;. -Djava.library.path=. Example</span></pre>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">
</span>
<br />
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">assuming the examle.jar and shared library are in your current folder. Note that Linux or OS X users will need to use a : as the classpath separator rather than ; as is required on Windows. But that's it! You just made a Java interface to your C++ library. You might have noticed the include(cmake_swig_jni) statement though. That is a bunch of CMake magic I had to write to make all this work, but work it does and on different platforms without trouble. You can see a larger example of a Java to C++ binding in <a href="https://github.com/mit-nlp/MITIE/tree/master/mitielib/java">MITIE's github repo</a> using this same setup.</span></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">
</span>
<br />
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">
</span></div>
Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com12tag:blogger.com,1999:blog-6061887630060661987.post-41590871079198118972014-08-28T22:23:00.000-04:002014-09-03T18:17:03.721-04:00Real-Time Face Pose EstimationI just posted the next version of dlib, <a href="http://sourceforge.net/projects/dclib/files/dlib/v18.10/dlib-18.10.tar.bz2">v18.10</a>, and it includes a number of new minor features. The main addition in this release is an implementation of an excellent paper from this year's Computer Vision and Pattern Recognition Conference:<br />
<blockquote class="tr_bq">
<span style="background-color: white;">One Millisecond Face Alignment with an Ensemble of Regression Trees by Vahid Kazemi and Josephine Sullivan</span></blockquote>
As the name suggests, it allows you to perform face pose estimation very quickly. In particular, this means that if you give it an image of someone's face it will add this kind of annotation:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioqEpgrNl36xhm7cllEZ-kE5xF-m806wV7TX9RvAwO53zEHeMLNBSuzgCnA8iyFA7OQp662HnqAuP_YvkJBQmJV59WIBeuS-rAxMjYofGkuocZtWXbtLGQ3lPxOuzJrY9lgbIVheMmtNw/s1600/landmarked_face2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioqEpgrNl36xhm7cllEZ-kE5xF-m806wV7TX9RvAwO53zEHeMLNBSuzgCnA8iyFA7OQp662HnqAuP_YvkJBQmJV59WIBeuS-rAxMjYofGkuocZtWXbtLGQ3lPxOuzJrY9lgbIVheMmtNw/s1600/landmarked_face2.png" height="396" width="400" /></a></div>
<br />
In fact, this is the output of dlib's new <a href="http://dlib.net/face_landmark_detection_ex.cpp.html">face landmarking example program</a> on one of the images from the HELEN dataset. To get an even better idea of how well this pose estimator works take a look at this video where it has been applied to each frame:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen='allowfullscreen' webkitallowfullscreen='webkitallowfullscreen' mozallowfullscreen='mozallowfullscreen' width='320' height='266' src='https://www.youtube.com/embed/oWviqXI9xmI?feature=player_embedded' frameborder='0'></iframe></div>
<br />
It doesn't just stop there though. You can use this technique to make your own custom pose estimation models. To see how, take a look at the <a href="http://dlib.net/train_shape_predictor_ex.cpp.html">example program</a> for training these pose estimation models.Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com330tag:blogger.com,1999:blog-6061887630060661987.post-30949601106709248482014-07-10T16:02:00.000-04:002014-07-10T16:14:10.382-04:00MITIE v0.2 Released: Now includes Python and C++ APIs for named entity recognition and binary relation extractionA few months ago I <a href="http://blog.dlib.net/2014/04/mitie-completely-free-and-state-of-art.html">posted about MITIE</a>, the new <a href="https://github.com/mit-nlp/MITIE">DARPA funded information extraction tool</a> being created by our team at MIT. At the time it only provided English named entity recognition and sported a simple C API. Since then we have been busy adding new features and today we released a new version of MITIE which adds a bunch of nice things, including:<br />
<ul>
<li>Python and C++ APIs</li>
<li>Many <a href="https://github.com/mit-nlp/MITIE/tree/master/examples">example programs</a></li>
<li>21 English binary relation extractors which identify pairs of entities with certain relations. E.g. "PERSON BORN_IN PLACE"</li>
<li>Python, C, and C++ APIs for training your own named entity and binary relation extractors</li>
</ul>
<div>
You can get MITIE from its <a href="https://github.com/mit-nlp/MITIE">github</a> page. Then you can try out some of the new features in v0.2, one of which is binary relation extraction. This means you can ask MITIE if two entities participate in some known relationship, for example, you can ask if a piece of text is making the claim that a person was born in a location. I.e. Are the person and location entities participating in the "born in" relationship? <br />
<br />
In particular, you could run MITIE over all the Wikipedia articles that mention Barack Obama and find each instance where someone made the claim that Barack Obama was born in some place. I did this with MITIE and found the following:<br />
<br />
<ul>
<li>14 claims that Barack Obama was born in Hawaii</li>
<li>5 claims that Barack Obama was born in the United States</li>
<li>3 claims that Barack Obama was born in Kenya</li>
</ul>
<div>
<br /></div>
<div>
Which is humorous. One of them is the sentence:</div>
<blockquote class="tr_bq">
You can still find sources of that type which still assert that "Barack Obama was born in Kenya"</blockquote>
<div>
When you read it in the broader context of the article it's clear that it's not claiming he was born in Kenya. So this is a good example of why it's important to aggregate over many relation instances when using a relation extractor. By aggregating many examples we can get reasonably accurate outputs in the face of these kinds of mistakes. </div>
<div>
<br /></div>
<div>
However, what is even more entertaining than poking fun at American political dysfunction is MITIE's new API for creating your own entity and relation extractors. We worked to make this very easy to use, and in particular, there are no parameters you need to mess with, everything is dealt with internal to MITIE. All you, the user, need to do is give example data showing what you want MITIE to learn to detect and it takes care of the rest. Moreover, in the spirit of easy to use APIs, we also added a new Python API that allows you to exercise all the functionality in MITIE via Python. As a little example, here is how you use it to find named entities:</div>
</div>
<pre class="code_box">from mitie import *
ner = named_entity_extractor('MITIE-models/english/ner_model.dat')
tokens = tokenize("The MIT Information Extraction (MITIE) tool was created \
by Davis King, Michael Yee, and Wade Shen at the \
Massachusetts Institute of Technology.")
print tokens</pre>
<div>
This loads in the English named entity recognizer model that comes with MITIE and then tokenizes the sentence. So the print statement produces </div>
<div>
<blockquote class="tr_bq">
['The', 'MIT', 'Information', 'Extraction', '(', 'MITIE', ')', 'tool', 'was', 'created', 'by', 'Davis', 'King', ',', 'Michael', 'Yee', ',', 'and', 'Wade', 'Shen', 'at', 'the', 'Massachusetts', 'Institute', 'of', 'Technology', '.']</blockquote>
</div>
<div>
Then to find the named entities we simply do</div>
<pre class="code_box">entities = ner.extract_entities(tokens)
print "Number of entities detected:", len(entities)
print "Entities found:", entities</pre>
<div>
Which prints:<br />
<blockquote class="tr_bq">
Number of entities detected: 6<br />
Entities found: [(xrange(1, 4), 'ORGANIZATION'), (xrange(5, 6), 'ORGANIZATION'), (xrange(11, 13), 'PERSON'), (xrange(14, 16), 'PERSON'), (xrange(18, 20), 'PERSON'), (xrange(22, 26), 'ORGANIZATION')]</blockquote>
</div>
<div>
So the output is just a list of ranges and labels. Each range indicates which tokens are part of that entity. To print these out in a nice list we would just do</div>
<pre class="code_box">for e in entities:
range = e[0]
tag = e[1]
entity_text = " ".join(tokens[i] for i in range)
print tag + ": " + entity_text</pre>
<div>
Which prints:<br />
<blockquote class="tr_bq">
ORGANIZATION: MIT Information Extraction<br />
ORGANIZATION: MITIE<br />
PERSON: Davis King<br />
PERSON: Michael Yee<br />
PERSON: Wade Shen<br />
ORGANIZATION: Massachusetts Institute of Technology</blockquote>
</div>
<div>
<a href="https://github.com/mit-nlp/MITIE">Now go give the new MITIE a try</a>!</div>
Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com12tag:blogger.com,1999:blog-6061887630060661987.post-17987350306677233412014-04-09T21:54:00.002-04:002014-04-10T06:43:12.142-04:00Dlib 18.7 released: Make your own object detector in Python!A while ago I <a href="http://blog.dlib.net/2014/02/dlib-186-released-make-your-own-object.html">boasted</a> about how dlib's object detection tools are better than OpenCV's. However, one thing OpenCV had on dlib was a nice Python API, but no longer! The new version of dlib is out and it includes a Python API for using and creating object detectors. What does this API look like? Well, lets start by imagining you want to detect faces in this image:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgy3W21GVtYHebnbmmluBQVFc9wBWL5M0Z2u93qVp1cQwAjCU4UANQk2jc5xZ3bqwNcRROy1q-rUckhD4e7vWXi2YQxPxlXnEr_90-Q70LEVknUh1gNOK4kNf-xn7kgYFFbIJw2KKDGsjc/s1600/WHITEHOUSEBEER.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgy3W21GVtYHebnbmmluBQVFc9wBWL5M0Z2u93qVp1cQwAjCU4UANQk2jc5xZ3bqwNcRROy1q-rUckhD4e7vWXi2YQxPxlXnEr_90-Q70LEVknUh1gNOK4kNf-xn7kgYFFbIJw2KKDGsjc/s1600/WHITEHOUSEBEER.jpg" height="290" width="400" /></a></div>
<br />
You would begin by importing dlib and <a href="http://scikit-image.org/">scikit-image</a>:<br />
<pre class="code_box">import dlib
from skimage import io</pre>
Then you load dlib's default face detector, the image of Obama, and then invoke the detector on the image:
<br />
<pre class="code_box">detector = dlib.get_frontal_face_detector()
img = io.imread('obama.jpg')
faces = detector(img)</pre>
The result is an array of boxes called faces. Each box gives the pixel coordinates that bound each detected face. To get these coordinates out of faces you do something like:
<br />
<pre class="code_box">for d in faces:
print "left,top,right,bottom:", d.left(), d.top(), d.right(), d.bottom()
</pre>
We can also view the results graphically by running:
<br />
<pre class="code_box">win = dlib.image_window()
win.set_image(img)
win.add_overlay(faces)
</pre>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfAz4oakaZUkXwnc3KFZ6BkE5FN7W4uwv7TmgkrAqxiJG9aKoCcnOyUUzIn15v109ckGfOj1NdjNH_Xza0b9PcVBgc3upPtneWrDwMZdlSN1w9E6DGiwyzv93sUoO8rzkCFkNIwzsQo-s/s1600/obama.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfAz4oakaZUkXwnc3KFZ6BkE5FN7W4uwv7TmgkrAqxiJG9aKoCcnOyUUzIn15v109ckGfOj1NdjNH_Xza0b9PcVBgc3upPtneWrDwMZdlSN1w9E6DGiwyzv93sUoO8rzkCFkNIwzsQo-s/s1600/obama.jpg" height="290" width="400" /></a></div>
<br />
But what if you wanted to create your own object detector? That's easy too. Dlib comes with an <a href="http://dlib.net/train_object_detector.py.html">example program</a> and a sample training dataset showing how to this. But to summarize, you do:<br />
<pre class="code_box">options = dlib.simple_object_detector_training_options()
options.C = 5 # Set the SVM C parameter to 5.
dlib.train_simple_object_detector("training.xml","detector.svm", options)
</pre>
That will run the trainer and save the learned detector to a file called detector.svm. The training data is read from training.xml which contains a list of images and bounding boxes. The example that comes with dlib shows the format of the XML file. There is also a graphical tool included that lets you mark up images with a mouse and save these XML files. Finally, to load your custom detector you do:
<pre class="code_box">detector = dlib.simple_object_detector("detector.svm")
</pre>
If you want to try it out yourself you can download the <a href="https://sourceforge.net/project/platformdownload.php?group_id=130373">new dlib release here</a>.Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com60tag:blogger.com,1999:blog-6061887630060661987.post-84778516489464158302014-04-03T22:45:00.001-04:002014-07-10T16:10:19.197-04:00MITIE: A completely free and state-of-the-art information extraction toolI work at a <a href="http://web.mit.edu/">MIT</a> lab and there are a lot of cool things about my job. In fact, I could go on all day about it, but in this post I want to talk about one thing in particular, which is that we recently got funded by the <a href="http://www.darpa.mil/OpenCatalog/XDATA.html">DARPA XDATA</a> program to make an open source natural language processing library focused on information extraction. <br />
<br />
Why make such a thing when there are already open source libraries out there for this (e.g. OpenNLP, NLTK, Stanford IE, etc.)? Well, if you look around you quickly find out that everything which exists is either expensive, not state-of-the-art, or GPL licensed. If you wanted to use this kind of NLP tool in a non-GPL project then you are either out of luck, have to pay a lot of money, or settle for something of low quality. Well, not anymore! We just released the first version of our <a href="https://github.com/mit-nlp/MITIE">MIT Information Extraction library</a> which is built using <a href="https://github.com/mit-nlp/MITIE/wiki/Evaluation">state-of-the-art</a> statistical machine learning tools. <br />
<br />
At this point it has just a <a href="https://github.com/mit-nlp/MITIE/blob/master/mitielib/include/mitie.h">C API</a> and an <a href="https://github.com/mit-nlp/MITIE/blob/master/examples/C/ner/ner_example.c">example program</a> showing how to do English named entity recognition. Over the next few weeks we will be adding bindings for other languages like Pyhton and Java. We will also be adding a lot more NLP tools in addition to named entity recognition, starting with relation extractors and part of speech taggers. But in the meantime you can use the C API or the streaming command line program. For example, if you had the following text in a file called sample_text.txt:<br />
<blockquote class="tr_bq">
Meredith Vieira will become the first woman to host Olympics primetime coverage on her own when she fills on Friday night for the ailing Bob Costas, who is battling a continuing eye infection. </blockquote>
Then you can simply run:<br />
<pre class="code_box">cat sample_text.txt | ./ner_stream MITIE-models/ner_model.dat
</pre>
And you get this as output:<br />
<blockquote class="tr_bq">
[PERSON Meredith Vieira] will become the first woman to host [MISC Olympics] primetime coverage on her own when she fills on Friday night for the ailing [PERSON Bob Costas] , who is battling a continuing eye infection .
</blockquote>
<div>
It's all up on <a href="https://github.com/mit-nlp/MITIE">github</a> so if you want to try it out yourself then just run these commands and off you go:</div>
<pre class="code_box">git clone https://github.com/mit-nlp/MITIE.git
cd MITIE
./fetch_submodules.sh
make examples
make MITIE-models
cat sample_text.txt | ./ner_stream MITIE-models/ner_model.dat</pre>
Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com23tag:blogger.com,1999:blog-6061887630060661987.post-66514595539033474522014-02-03T22:55:00.000-05:002015-02-04T03:24:52.192-05:00Dlib 18.6 released: Make your own object detector!<div class="separator" style="clear: both; text-align: left;">
I just posted the next version of dlib, v18.6. There are a bunch of nice changes, but the most exciting addition is a tool for creating histogram-of-oriented-gradient (HOG) based object detectors. This is a technique for detecting semi-rigid objects in images which has become a classic computer vision method since its publication in 2005. In fact, the original HOG paper has been cited <a href="http://scholar.google.com/scholar?q=Histograms+of+Oriented+Gradients+for+Human+Detection+by+Navneet+Dalal+and+Bill+Triggs">over 7000 times</a>, which for those of you who don't follow the academic literature, is a whole lot.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
But back to dlib, the new release has a tool that makes training HOG detectors super fast and easy. For instance, here is an example program that shows <a href="http://dlib.net/fhog_object_detector_ex.cpp.html">how to train a human face detector</a>. All it needs as input is a set of images and bounding boxes around faces. On my computer it takes about 6 seconds to do its training using the example face data provided with dlib. Once finished it produces a HOG detector capable of detecting faces. An example of the detector's output on a new image (i.e. one it wasn't trained on) is shown below:</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaq6urV3lcpNGJyQs31I5zk-TVFrKvPhwgaB4j5Y39-JMH3xC2jJxv2F_AtDJI7f1Ua_uz5HojKUzdL5xkqgL6t-ZbxYn_eHpz8cYZFXF2LljPt5bzHJEcTMXrNoJ9GruA0lJY7GgGsUM/s1600/face_detections.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaq6urV3lcpNGJyQs31I5zk-TVFrKvPhwgaB4j5Y39-JMH3xC2jJxv2F_AtDJI7f1Ua_uz5HojKUzdL5xkqgL6t-ZbxYn_eHpz8cYZFXF2LljPt5bzHJEcTMXrNoJ9GruA0lJY7GgGsUM/s1600/face_detections.jpg" height="424" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
You should compare this to the time it takes to train OpenCV's popular cascaded haar object detector, which is generally reported to take <a href="http://stackoverflow.com/questions/19487303/how-much-time-does-opencv-haar-training-take">hours</a> or <a href="http://answers.opencv.org/question/757/haartraining-vs-traincascade-object-detection/">days</a> to train and requires you to <a href="http://note.sonots.com/SciSoftware/haartraining.html">fiddle with false negative rates</a> and all kinds of spurious parameters. HOG training is considerably simpler.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Moreover, the HOG trainer uses dlib's <a href="http://arxiv.org/abs/1502.00046">structural SVM based training algorithm</a> which enables it to train on all the sub-windows in every image. This means you don't have to perform any tedious subsampling or "hard negative mining". It also means you often don't need that much training data. In particular, the <a href="http://dlib.net/fhog_object_detector_ex.cpp.html">example program that trains a face detector</a> takes in only 4 images, containing a total of 18 faces. That is sufficient to produce the HOG detector used above. The example also shows you how to visualize the learned HOG detector, which in this case looks like:</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRUuYwXYbCdNiCJeXij7c0LWqfLYKK11m62sZczNm0BQTzNMQ3FYK0O1TJ8RkzO2272B6ITUHVkvuVjovKhyphenhyphenTlLV6Xua7jI3HoX1z3oawsuBtKb0XhnQSImN1yu5GvGPVglpSU7zo0cqQ/s1600/face_fhog_filters.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRUuYwXYbCdNiCJeXij7c0LWqfLYKK11m62sZczNm0BQTzNMQ3FYK0O1TJ8RkzO2272B6ITUHVkvuVjovKhyphenhyphenTlLV6Xua7jI3HoX1z3oawsuBtKb0XhnQSImN1yu5GvGPVglpSU7zo0cqQ/s1600/face_fhog_filters.png" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
It looks like a face! It should be noted that it's worth training on more than 4 images since it doesn't take that long to label and train on at least a few hundred objects and it can improve the accuracy. In particular, I trained a HOG face detector using about 3000 images from the <a href="http://vis-www.cs.umass.edu/lfw/">labeled faces in the wild</a> dataset and the training took only about 3 minutes. 3000 is probably excessive, but who cares when training is so fast.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The face detector which was trained on the labeled faces in the wild data comes with the new version of dlib. You can see how to use it in this <a href="http://dlib.net/face_detection_ex.cpp.html">face detection example program</a>. The underlying detection code in dlib will make use of SSE instructions on Intel CPUs and this makes dlib's HOG detectors run at the same speed as OpenCV's fast cascaded object detectors. So for something like a 640x480 resolution web camera it's fast enough to run in real-time. As for the accuracy, it's easy to get the same detection rate as OpenCV but with <i style="font-weight: bold;">thousands of times fewer false alarms</i>. You can see an example in this youtube video which compares OpenCV's face detector to the new HOG face detector in dlib. The circles are from OpenCV's default face detector and the red squares are dlib's HOG based face detector. The difference is night and day. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div style="text-align: center;">
<iframe allowfullscreen='allowfullscreen' webkitallowfullscreen='webkitallowfullscreen' mozallowfullscreen='mozallowfullscreen' width='320' height='266' src='https://www.youtube.com/embed/LsK0hzcEyHI?feature=player_embedded' frameborder='0'></iframe></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: left;">
</div>
Finally, here is another fun example. Before making this post I downloaded 8 images of stop signs from Google images, drew bounding boxes on them and then trained a HOG detector. This is the detector I got after a few seconds of training:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfNaQvrwNLOwNP5jqgUEY8jtvjUz6ux9EGZLsU9XF1VPvCi3lc-dmeQAmqrD2D47aMrTN06BKJd0GVFZtQrEXh-MV1p3m6F1hM3P46nbL-b6ZsItn1DNJLHrGY8ZoDTCgoU6efJsUA2bs/s1600/stopsign_fhog_filter.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfNaQvrwNLOwNP5jqgUEY8jtvjUz6ux9EGZLsU9XF1VPvCi3lc-dmeQAmqrD2D47aMrTN06BKJd0GVFZtQrEXh-MV1p3m6F1hM3P46nbL-b6ZsItn1DNJLHrGY8ZoDTCgoU6efJsUA2bs/s1600/stopsign_fhog_filter.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
It looks like a stop sign and testing it on a new image works great.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQjkIV8Qm5VhRtLgnQATobVp7ARHEYX-54etkwSOTD0rLPI57Nb3MtmpKID3r1K0wiTpL6Q3CzBHR7sJX9jdXT4Rw__STDqeVwVKjCsOQ4w9kBE3CvVBZaITaM1vZSTbNbA520CR-qvGw/s1600/test_stopsign.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQjkIV8Qm5VhRtLgnQATobVp7ARHEYX-54etkwSOTD0rLPI57Nb3MtmpKID3r1K0wiTpL6Q3CzBHR7sJX9jdXT4Rw__STDqeVwVKjCsOQ4w9kBE3CvVBZaITaM1vZSTbNbA520CR-qvGw/s1600/test_stopsign.png" height="400" width="297" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
All together it took me about 5 minutes to go from not having any data at all to a working stop sign detector. Not too shabby. Go try it out yourself. You can get the <a href="https://sourceforge.net/project/platformdownload.php?group_id=130373">the new dlib release here</a> :)Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com252tag:blogger.com,1999:blog-6061887630060661987.post-9564356532419725632007-03-09T19:10:00.000-05:002013-07-04T13:29:29.899-04:00Adding a web interface to a C++ applicationOne thing that is always sort of a pain is setting up a graphical user interface. This is especially true if you are making an embedded application or something that functions more as a system service or daemon. In this case you probably end up creating some simple network protocol which you use to control your application via some other remote piece of software or just telnet if you are feeling especially lazy.<br />
<br />
What would be nice, however, is to have a web based interface but not have to jump through a bunch of hoops to add it into your code. This sort of simple add-in HTTP server is exactly what this post is all about.<br />
<br />
A few months ago I was spending Christmas with the family. Good times all around but they live out in the middle of nowhere. And when I mean nowhere I'm talking no TV, 1 bar on the cell phone during a 'clear' conversation, no internet, and some farm animals. We do have what is seemingly an ex-circus goat that walks around by balancing on just its front two feet. You can imagine how entertaining that is but even so it only occupies you for so long. So naturally I broke out my laptop and created just the thing I have always wanted, a simple HTTP server I can stick into my C++ applications. I love Christmas :)<br />
<br />
The code for the thing is available from <a href="http://dclib.sf.net/">sourceforge</a>. There is also a more involved example program that can be viewed in its entirety <a href="http://dclib.sourceforge.net/server_http_ex.cpp.html">here</a>, but for the sake of creating a somewhat tidy little tutorial I'll show a simple example.<br />
<pre style="background: rgb(238, 238, 238);"><span style="color: blue;">#include</span> <span style="color: #5555ff;"><</span>dlib<span style="color: #5555ff;">/</span>server.h<span style="color: #5555ff;">></span>
<span style="color: blue;">using</span> <span style="color: blue;">namespace</span> dlib;
<span style="color: blue;">class</span> <b><a href="" name="web_server"></a>web_server</b> : <span style="color: blue;">public</span> server_http
<b>{</b>
<span style="color: blue;">const</span> std::string <b><a href="" name="on_request"></a>on_request</b> <span style="font-family: Lucida Console;">(</span>
<span style="color: blue;">const</span> incoming_things<span style="color: #5555ff;">&</span> incoming,
outgoing_things<span style="color: #5555ff;">&</span> outgoing
<span style="font-family: Lucida Console;">)</span>
<b>{</b>
<span style="color: blue;">return</span> "<span style="color: #cc0000;"><html><body>Hurray for simple things!</body></html></span>";
<b>}</b>
<b>}</b>;
<span style="color: blue;"><u>int</u></span> <b><a href="" name="main"></a>main</b><span style="font-family: Lucida Console;">(</span><span style="font-family: Lucida Console;">)</span>
<b>{</b>
web_server our_web_server;
our_web_server.<span style="color: #bb00bb;">set_listening_port</span><span style="font-family: Lucida Console;">(</span><span style="color: #979000;">80</span><span style="font-family: Lucida Console;">)</span>;
our_web_server.<span style="color: #bb00bb;">start</span><span style="font-family: Lucida Console;">(</span><span style="font-family: Lucida Console;">)</span>;
<b>}</b>
</pre>
Basically what is happening here, if it isn't already obvious, is we are defining a class which acts as our HTTP server. To do this all you need to do is inherit from server_http and implement the virtual function on_request(). To turn it on just set the listening port number and call start(). That's it. If you compiled this and ran it you could check out the page it creates by directing your browser to http://localhost/ and it would pop up. You would see a page with the single line of text "Hurray for simple things!".<br />
<br />
For an explanation of the arguments to the on_request() function check out the example program linked to above and/or check out the documentation on the dlib web site.Davis Kinghttp://www.blogger.com/profile/16577392965630448489noreply@blogger.com0