How do I extract specific text from an image using a UITextField in Swift?

Question

I am using the Vision framework and I want to be able to use a UITextField to find a specific word in a picture. For example let's say I type in the word black in the text field and I want it to detect that in the picture I have. How would I do that? Im using Vision framework and I figured out how to detect the texts but stuck on the part where I can detect the user inputed word in the text field.

        func startTextDetection() {

       let textRequest = VNDetectTextRectanglesRequest(completionHandler: self.detectTextHandler)
       let request = VNRecognizeTextRequest(completionHandler: self.detectTextHandler)

        request.recognitionLevel = .fast
        textRequest.reportCharacterBoxes = true
        self.requests = [textRequest]

    }

    func detectTextHandler(request: VNRequest, error: Error?) {
        guard let observations = request.results else {
            print("no result")
            return
        }

        let result = observations.map({$0 as? VNTextObservation})

        DispatchQueue.main.async() {
            self.previewView.layer.sublayers?.removeSubrange(1...)
            for region in result {
                guard let rg = region else {
                    continue
                }

                self.highlightWord(box: rg)
                if let boxes = region?.characterBoxes {
                    for characterBox in boxes {
                        self.highlightLetters(box: characterBox)
                }
            }
        }
    }
}

     //when user presses search will search for text in pic. 
func textFieldShouldReturn(_ searchTextField: UITextField) -> Bool {
    searchTextField.resignFirstResponder()
    startTextDetection()

    return true
}

Check out [this thread](https://stackoverflow.com/questions/44533148/converting-a-vision-vntextobservation-to-a-string), there are a couple of answers that might help you. — Ivan Nesterenko, Feb 24 '20 at 23:02

score 4 · Accepted Answer · answered Feb 25 '20 at 09:45

You should watch the latest WWDC on Vision framework. Basically, from iOS 13 the VNRecognizeTextRequest returns the text and also the bounding box of the text in the image. The code can be something like this:

func startTextDetection() {
    let request = VNRecognizeTextRequest(completionHandler: self.detectTextHandler)
    request.recognitionLevel = .fast
    self.requests = [request]
}

private func detectTextHandler(request: VNRequest, error: Error?) {
    guard let observations = request.results as? [VNRecognizedTextObservation] else {
        fatalError("Received invalid observations")
    }
    for lineObservation in observations {
        guard let textLine = lineObservation.topCandidates(1).first else {
            continue
        }

        let words = textLine.string.split{ $0.isWhitespace }.map{ String($0)}
        for word in words {
            if let wordRange = textLine.string.range(of: word) {
                if let rect = try? textLine.boundingBox(for: wordRange)?.boundingBox {
                     // here you can check if word == textField.text
                     // rect is in image coordinate space, normalized with origin in the bottom left corner
                }
            }
        }
   }
}

Hey thanks for help but how would I show the bounding box over the word when I search for it in the UITextField? I know the word is getting detected because I used the `If statement` and checked it using the `print(word)`. I just want to be able to highlight the word that was searched for. Thanks! — xcode22, Feb 26 '20 at 16:20
The code I've posted also has the `CGRect` (`rect` variable) representing the location of the text. But that `CGRect` has the coordinates normalized to the dimensions of the processed image, with the origin in the lower-left corner. In order to covert it from image space to your `UIImageView` coordinate space you should check out `VNImageRectForNormalizedRect` and [ImageCoordinateSpace](https://github.com/paulz/ImageCoordinateSpace). Once you've converted the `CGRect`, create a view with that frame, give it a border width and colour and add it as a subview to your `UIImageView`. — Costea Bogdan, Feb 27 '20 at 12:08

How do I extract specific text from an image using a UITextField in Swift?

1 Answers1