I agree that this is not an easy thing to do in MySQL. If your aiming to output the result into the web app, it's probably much easier to obtain the necessary information from MySQL and figure out how to do it on a web app. That being said, I did actually manage to get the result using a combination of LAG(), LEAD() and a quite complicated CASE expression:
SELECT acc_name,
installment_amt,
installment_no,
paid_amt,
CASE WHEN paid_amt > 0
AND paid_amt+nxp >= installment_amt
AND diff > 0
THEN 0
WHEN paid_amt > 0
AND paid_amt+nxp < installment_amt
AND nxp > 0
THEN nxp
WHEN paid_amt > 0
AND paid_amt+nxp < installment_amt
AND LAG(diff) OVER (PARTITION BY acc_name ORDER BY installment_no) > 0
THEN LAG(installment_amt-diff) OVER (PARTITION BY acc_name ORDER BY installment_no)
WHEN paid_amt > 0
AND paid_amt+nxp < installment_amt
AND diff <0
THEN ABS(diff)+paid_amt
ELSE 0
END balance
FROM
(SELECT *,
CASE WHEN paid_amt > 0
AND paid_amt+nxp >= installment_amt
THEN paid_amt+nxp-installment_amt
WHEN paid_amt > 0
AND paid_amt+nxp < installment_amt
THEN (paid_amt+nxp)-installment_amt
END AS diff
FROM
(SELECT *,
CASE WHEN paid_amt < installment_amt THEN
LEAD(paid_amt) OVER (PARTITION BY acc_name ORDER BY installment_no)
ELSE 0 END nxp
FROM installment) V
) R
The first step is to get the next paid_amt using LEAD() with a CASE expression. Then make that as a subquery. The next step is to use that LEAD() result from the subquery to do some simple calculation but with a slightly more advanced CASE conditions. Once that done, turn it into a subquery. The last step have a much more complicated CASE expression that I use with LAG(). By this point, I already managed generate two columns; nxp - is the next paid_amt that was obtained in the first subquery and diff - is the difference calculated in the second subquery.
I don't know how well this query will work if there's other variation of data than the one you've mentioned but at the moment, it does return the desired result.
Here's a demo fiddle