Step 1 :
Factor out common factor $ \color{blue}{ 10xy } $:
$$ 10x^5y-40xy^3 = 10xy ( x^4-4y^2 ) $$Step 2 :
Rewrite $ x^4-4y^2 $ as:
$$ \color{blue}{ x^4-4y^2 = (x^2)^2 - (2y)^2 } $$Now we can apply the difference of squares formula.
$$ I^2 - II^2 = (I - II)(I + II) $$After putting $ I = x^2 $ and $ II = 2y $ , we have:
$$ x^4-4y^2 = (x^2)^2 - (2y)^2 = ( x^2-2y ) ( x^2+2y ) $$