This solution below will iterate through each cell with a value in column B of sheet1 and check it against every value in Column A of sheet89 (although you named this ColumnO, according to your getValues function, it will grab values from Column A).
If it finds a match, it will turn green the cell in column B of sheet1.  In your example code you use the i loop variable (which iterates through rows on sheet89) to get the cell on sheet1 to turn green.  It's not clear which cells you want to turn green.  I assumed it was the cells on sheet1 so I changed the the code to
sheet.getRange(h+1, 2).setBackgroundColor('green');
Also, the getRange function for a single cell only requires 2 arguments, so I removed the numRows and numColumns arguments for the line which colors the cell green.
I'm not sure why bValues and oValues exclude the last row, but I removed the -1 in each of these as it will cause the code to fail if for any reason it is run on a blank worksheet.  The getLastRow() returns the last row with a value, not the next blank row in the sheet.  If you want to capture the whole sheet, then you shouldn't use -1.
function formatting() {
  var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Sheet1');
  var columnB = sheet.getRange(1, 2, sheet.getLastRow(), 1);
  var bValues = columnB.getValues();
  var sheet89 = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Sheet89');
  var columnO = sheet89.getRange(1, 1, sheet.getLastRow(), 1);
  var oValues = columnO.getValues();
  for (var h = 0; h < bValues.length; h++) {
     for (var i = 0; i < oValues.length; i++) {
       if (oValues[i][0] == bValues[h][0]) {
         sheet.getRange(h + 1, 2).setBackgroundColor('green');
       }  
     }
  }
}