Step 1 :
Factor out common factor $ \color{blue}{ -5n } $:
$$ -5m^3n+40n^4 = -5n ( m^3-8n^3 ) $$Step 2 :
To factor $ m^{3}-8n^{3} $ we can use difference of cubes formula:
$$ I^3 - II^3 = (I - II) (I^2 + I \cdot II + II^2) $$After putting $ I = m $ and $ II = 2n $ , we have:
$$ m^{3}-8n^{3} = ( m-2n ) ( m^{2}+2mn+4n^{2} ) $$