Step 1 :
Factor out common factor $ \color{blue}{ ab } $:
$$ a^2b^2-ab^3+2a^3b = ab ( ab-b^2+2a^2 ) $$Step 2 :
$ ab-b^2+2a^2 $ can be written as
$$ ab-b^2+2a^2 = \left( 2a-b \right) \left( a+b \right)$$The explanation for this step cannot be generated.