Tag: limits

I wanted to cover the topic of limits in calculus. Calculus is often called the mathematics of change, and handles topics like rates of change. These are super important topics with tons of applications, especially within machine learning, deep learning and AI in general. I’ll build on this topic of limits in the following posts on derivatives and integrals; topics that will be fundamental when I cover Deep Learning from first principles at some later date. Let’s get stuck right in 🙂

Limits – take it to infinity!

First some background. Limits are really a central idea in calculus. In plain English it goes something like this: think of a number (let’s say the number 4), now realise there there are infinitely many decimal numbers either side of this number (e.g. 3.9999999999 and 4.0000000000001). If we keep making these decimals get longer and longer those numbers both edge closer and closer to the whole number (4) without ever quite reaching it. If we could keep going infinitely we’d eventually reach 4 (kinda), right? We would say that 4 is the limit. Okay, it’s not quite like that, but at least that’s the basic idea.

3.999999999 >>> 4 <<< 4.00000001

So why is this useful? Well it means that in any situation where some rate of a function is changing, we can figure out what the exact rate is at a given moment. Okay, that sounds confusing so let’s make it more concrete using an example: let’s say we find a dark, dried up well and we want to know how deep it is. We can’t see the bottom so we decide to drop a stone in and count how long it takes before we hear the stone bounce. We have the formula for how far the stone has fallen after a given number of seconds t:

d(t) = 0.5 \times 9.81 \times t^2

Where d(t) is the distance in metres the ball has fallen after t seconds. After we drop the stone, we hear it hit the bottom after 3.5 seconds. So the stone fell about 60 metres! We can calculate the average speed of the stone by dividing the distance by the time (where y is the distance and t is the time at two observations):

\begin{aligned}
&average \space speed = \frac{distance}{time}= \frac{y_2 - y_1}{t_2 - t_1}
\end{aligned}

Which after plugging in the numbers we discover the stone fell at about 17.14m/s! But this is an average. If we drew a graph for our function d(t)=0.5 \times 0.981 \times t^2 , this approach would be equivalent to drawing a straight line from 0 to 3.5 on the graph and calculating the good old “rise over run” to get the slope. In this instance the slope is our speed. See for yourself below:

The green secant line (intersects the curve in two places) connects the points (0,0) and (3.5, 60). The slope of the green line is the average speed during this time: 60.8/3.5 = 17.14! It’s good to keep this visualisation in mind for when we cover derivatives – we are drawing a straight line and using it to approximate the actual observed curved line. We will do almost exactly the same thing with derivatives later on!

We can use this approach to find the average speed at any given time range, for example how fast did the stone fall between 1 second and 3.5 seconds? We know that after 3.5 seconds the stone had fallen 60m, and using our formula for distance (or the graph above) we can see that the stone fell 4.9m after 1 second. So between 1 second and 3.5 seconds the stone fell 60 - 4.9 = 55.1 metres, over the course of 2.5 seconds (3.5 – 1):

Average \space speed = \frac{60 - 4.9}{3.5 - 1} = \frac{55.1}{2.5} = 22.4

Finding the ‘instantaneous’ speed:

Cool, so the average speed between 1 second and 3.5 seconds was 22.4 m/s! But can we apply the approach to find the exact speed at 1 second after the stone was dropped? Let’s try it:

Average \space speed = \frac{4.9 - 4.9}{1 - 1} = \frac{0}{0} = ?

Okay so \frac{0}{0} is undefined. Does this mean the speed at 1 second was nothing… ? You must be thinking that cannot be true, and you are right. This is where limits come in, let’s express this whole thing as a limit, and substitute our original distance equation in to the equation for speed:

Instantaneous \space speed =\lim_{t\to1} \frac{(0.5 \times 9.81 \times t^2) - (0.5 \times 9.81 \times 1^2)}{t - 1}

This formula is exactly the same as the one above for the average speed, where we take the difference between two distances at two different times to figure out how fast the stone was moving. We also substituted in some given time t, and also the distance at 1 second (because we want to find the speed of the stone at exactly 1 second, remember?).

You read the \lim_{t\to1} part like this: as t approaches 1. What does this mean? Think back to the start of this post where we pointed out that there are infinitely many numbers on either side of a given number, that’s what this \lim_{t\to1} is saying: that if t takes increasingly tiny steps closer and closer to 1 in the following expression, then what output value do we start to get closer to? You can imagine this visually using the same straight line on the graph of a curve we showed above, with one difference: imagine you are moving the two ends of the straight line (where it intersects the curve) closer and closer to 1. What does the slope of that straight line get closer and closer to as we move the ends closer and closer to 1? If we get the ends of this line as close as possible to 1, and zoom in really close, the curve will also look like a straight line, and the straight line we drew will be tangent to the curve. So we can use the slope of the straight line we drew to approximate the slope (a.k.a gradient) of the curve at this point! 🙂

We have a little more work to do before we can figure out what that speed value is at exactly 1 second though. We already tried substituting 1 for t in the equation Average \space speed = \frac{4.9 - 4.9}{1 - 1} = \frac{0}{0} = ? and it was undefined. So what do we do? The good news is we can do some good old algebra to make this work:

\begin{aligned}
Instantaneous \space speed &=\lim_{t\to1} \frac{(0.5 \times 9.81 \times t^2) - (0.5 \times 9.81 \times 1^2)}{t - 1}\\
&= \lim_{t\to1} \frac{(4.905 \times t^2) - (4.905 \times 1^2)}{t - 1}\\
&= \lim_{t\to1} \frac{4.905(t^2 -1^2)}{t - 1}\\
&= \lim_{t\to1} \frac{4.905(t-1)(t+1)}{t - 1}\\
&= \lim_{t\to1}{4.905(t+1)}\\
\end{aligned}

After a bit of simplification and factoring, now we have a form where we can plug in our value (1) and see what the ‘instantaneous’ speed of the stone is at exactly 1 second after we dropped it into the well:

= 4.905(1+1) = 4.905 \times 2 = 9.81

So all we need to do is simplify our function and that will make it possible for us to substitute the limit value into the function to find out what the output is. This works because our algebraic manipulation will plug the gap in the function where it is undefined.

So we figured it out, at one second after we dropped the stone, the speed of the falling stone is 9.81 metres per second!

Summing it all up:

Let’s take a moment to digest everything we just did. We figured out how deep our wishing well is. We realised that speed is a rate \frac{distance}{time}, and that we can use this fraction to figure out the average speed over any given distance and time. But we can’t use it to figure out the exact speed in a single moment. We realised that we can use limits to figure out how fast the stone is travelling at any moment, by gradually making t closer and closer to the time we wanted the speed for (1 second). We used some algebra to rearrange our formula for speed so that we could plug in the exact value (1 second) and find the exact speed of the stone one second after dropping it into the well!
We realised that we can think of this whole process as drawing a secant line across our function curve, and slowly moving that secant line closer and closer to our point of interest until it becomes a tangent line. The slope of this tangent line will be the same as the slope of the curve at that exact point.

Finding Limits – step by step process:

All of the above was just for context, this part is really all you need to find limits out in the wild:

1. We tried substituting our value of the limit (1) into the function, and we got \frac{0}{0} which is undefined.
2. We simplified and factored our function.
3. We plugged the value of our limit (1) back into the formula and got the speed of our stone at the exact point we were interested in!

We can follow this general approach to find pretty much any limit. You should really memorise this process. If a function contains a fraction with square roots the approach will be slightly different, we just need to rationalise the function by multiplying the numerator and denominator by the conjugate of the expression containing the square root (a conjugate is just the same expression with + and – switched) like this:

\begin{aligned}
&\lim_{x\to4}{\frac{\sqrt{x} -2}{x-4}}\\
&=\lim_{x\to4}{\frac{\sqrt{x} -2}{x-4} \times \frac{\sqrt{x}+2}{\sqrt{x}+2}}\\
&=\lim_{x\to4}{\frac{(\sqrt{x})^2 -2^2}{(x-4)(\sqrt{x} +2)}}\\
&=\lim_{x\to4}{\frac{x - 4}{(x-4)(\sqrt{x} +2)}}\\
&=\lim_{x\to4}{\frac{1}{\sqrt{x} +2}}\\
\end{aligned}

And now the substitution for x = 4 will work: \frac{1}{\sqrt{4}+2} = \frac{1}{4} 🙂

In some other cases you might need to do some other algebraic simplification/manipulation. You can try adding/subtracting/multiplying/dividing fractions, cancelling, etc. In the end, just try to simplify the function and then substitution should work. If all else fails, use this online tool to see a breakdown of how to simplify and find limits!