Ruby Invalid Byte Sequence in UTF-8

Question

I have the following code, which gives me an invalid byte sequence error pointing to the scan method in initialize. Any ideas on how to fix this? For what it's worth, the error does not occur when the (.*) between the h1 tag and the closing > is not there.

#!/usr/bin/env ruby

class NewsParser

  def initialize
      Dir.glob("./**/index.htm") do |file|
        @file = IO.read file 
        parsed = @file.scan(/<h1(.*)>(.*?)<\/h1>(.*)<!-- InstanceEndEditable -->/im)
        self.write(parsed)
      end
  end

  def write output
    @contents = output
    open('output.txt', 'a') do |f| 
      f << @contents[0][0]+"\n\n"+@contents[0][1]+"\n\n\n\n" 
    end
  end

end

p = NewsParser.new

Edit: Here is the error message:

news_parser.rb:10:in 'scan': invalid byte sequence in UTF-8 (ArgumentError)

SOLVED: The combination of using: @file = IO.read(file).force_encoding("ISO-8859-1").encode("utf-8", replace: nil) and encoding: UTF-8 solve the issue.

Thanks!

possible duplicate of [ruby 1.9: invalid byte sequence in UTF-8](http://stackoverflow.com/questions/2982677/ruby-1-9-invalid-byte-sequence-in-utf-8) — AShelly, Mar 07 '12 at 19:17
`@file = IO.read(file).force_encoding("ISO-8859-1").encode("utf-8", replace: nil)` ? — fl00r, Mar 07 '12 at 19:21
Try putting `#encoding: UTF-8` as the second line in your file. — Phrogz, Mar 07 '12 at 19:48
You should post your solution as an answer then accept it; it might help others in the future. — Jwosty, Mar 07 '12 at 23:16
For what it's worth the [Ruby docs state](http://ruby-doc.org/core-1.9.3/IO.html#method-i-read) that the stream read by IO#read is _always_ ASCII-8BIT. — Wolfram Arnold, Jan 29 '14 at 21:43

score 41 · Accepted Answer · answered Mar 08 '12 at 02:18

41

The combination of using: @file = IO.read(file).force_encoding("ISO-8859-1").encode("utf-8", replace: nil) and #encoding: UTF-8 solved the issue.

answered Mar 08 '12 at 02:18

redgem

1,453
4
15
28

score 1 · Answer 2 · answered Sep 14 '21 at 00:56

While this question already has an accepted answer, I found it while having the same problem with a different style of opening the file:

File.open(file_name).each_with_index do |line, index|
  line.gsub!(/[{}]/, "'")
  puts "#{index} #{line}"
end

I found that my input file was encoded in ISO-8859-1, so I changed it to the following to avoid the error:

File.open(file_name, 'r:ISO-8859-1:utf-8').each_with_index do |line, index|
  line.gsub!(/[{}]/, "'")
  puts "#{index} #{line}"
end

See the documentation for the optional mode argument of the File.open method for more details.

Ruby Invalid Byte Sequence in UTF-8

2 Answers2

Linked