Step 1 :
Factor out common factor $ \color{blue}{ 4ab } $:
$$ 4a^5b-64ab = 4ab ( a^4-16 ) $$Step 2 :
Rewrite $ a^4-16 $ as:
$$ \color{blue}{ a^4-16 = (a^2)^2 - (4)^2 } $$Now we can apply the difference of squares formula.
$$ I^2 - II^2 = (I - II)(I + II) $$After putting $ I = a^2 $ and $ II = 4 $ , we have:
$$ a^4-16 = (a^2)^2 - (4)^2 = ( a^2-4 ) ( a^2+4 ) $$Step 3 :
Rewrite $ a^2-4 $ as:
$$ \color{blue}{ a^2-4 = (a)^2 - (2)^2 } $$Now we can apply the difference of squares formula.
$$ I^2 - II^2 = (I - II)(I + II) $$After putting $ I = a $ and $ II = 2 $ , we have:
$$ a^2-4 = (a)^2 - (2)^2 = ( a-2 ) ( a+2 ) $$