0

I have a CSV file with 412.000 strings in that I would like to pre-store locally so that I can deploy to Android and iOS. The game must then be able to look through these strings to check if there's a match based on user input.

The only viable solution that I can see would be SQLite. I haven't come across a very good SQLite solution for Unity yet.

Is there a built-in solution in Unity that I am overlooking?

The solution has to work locally. No HTTP calls.

Doh09
  • 2,324
  • 1
  • 16
  • 31
  • How long are the strings, and how do you need to search them? (startswith, contains, in/sens etc) – Caius Jard Dec 23 '20 at 21:05
  • Less than 50 characters, most below 25. needs to be a 100% match. So if I write "ThisCanBeAStringExample" it needs to check if that is in the available strings. I was thinking maybe one could somehow load the CSV file into memory as a Dictionary but didn't figure out how yet. – Doh09 Dec 23 '20 at 21:07
  • Case sensitivity can be skipped, should not matter if it's upper or lower case actually. – Doh09 Dec 23 '20 at 21:08
  • Am not sure if the suggestion I made with a dictionary is very memory heavy or not, would have to test. – Doh09 Dec 23 '20 at 21:11
  • You could simply use a local Database, no need for external HTTP calls e.g. https://github.com/rizasif/sqlite-unity-plugin – derHugo Dec 23 '20 at 21:33
  • As I wrote I am aware that SQLite might be a viable solution but I have not come across a good version of that for Unity yet. The one in the link has not been updated in 2 years. It might still work but preferably i'd use one that is still being fitted for the newer versions of the Unity editor. Lots have happened in Unity since 2018. – Doh09 Dec 23 '20 at 21:46
  • Utterly no need for a databse for this. – Fattie Dec 24 '20 at 01:11
  • 1
    I see **no** reason to close vote this question - in fact it's an excellent question. – Fattie Dec 24 '20 at 01:13

1 Answers1

1

400,000 strings is absolutely trivial.

Just put them in a dictionary (list, whatever is relevant and that you prefer).

It's a total non-issue.

It's likely you would just load them from a text file, easy as pie.

 public TextAsset theTextFile;

(Just drag to the link in the Inspector, like any texture or similar.)

you can then very easily read that file as, say, JSON. (Just use JsonUtility. You can find numerous examples of this in SO and elsewhere.) For example,

   Blah bb = JsonUtility.FromJson< Blah >(ta.text);
   yourDict = bb.fieldname.ToDictionary(i => i.tag, i => i);

Note that you mention "memory" and so on. It's totally irrelevant, the data you are talking about is the fraction of the size of any tiny image - ! , it's a non-issue, you don't have to think about it. The hardware/software system will handle it.

P.S. ...

If you literally want to use csv, it's totally easy. I suggest you ask a new question giving the details of your file and so on, so you can get an exact answer.

Note that you'd just use a HashSet rather than a Dictionary. It's even easier.

It's just something like:

var wordList = theTextFile.text.Split('\n');

You can google many examples!

https://stackoverflow.com/a/9791488/294884

http://answers.unity.com/answers/397537/view.html

Fattie
  • 27,874
  • 70
  • 431
  • 719
  • I tried loading it into a dictionary a few days ago by using a StreamReader to read the lines from the CSV file. When I did so my program froze. You can see the code here: https://pastebin.com/4pkis4Eh. If I added a "maxTestAmount" that increased for each word added and set it to say 5000, then it froze in the beginning but unfroze after a while. Is the approach in the link not an effective one? – Doh09 Dec 25 '20 at 12:44
  • I believe it worked! I tried this time with the approach in this link: https://pastebin.com/EnnRg1kn and it loaded instantly. The count says +400k strings loaded in. – Doh09 Dec 25 '20 at 12:56
  • A new problem has arisen, it turns out that while the approach does load fast it doesn't load the Nordic characters æ, ø and å in correctly. It did resolve my original question though. – Doh09 Dec 25 '20 at 13:12
  • Using a brief python script I found that the encoding is set to 'cp1252' for the CSV file. Was expecting UTF8. Will try work around that. – Doh09 Dec 25 '20 at 13:26
  • If I use UTF8 it loads them as '�', if I use GetEncoding(1252) from System.Text.Encoding, then it makes them appear as '†'. I then found that the OpenOffice Calc program I had used for editing the CSV file opened it as "DOS/OS2-865/Nordic". So I tried getting 865 encoding, I had tried that earlier but this time I tried setting it as an integer instead of a string argument. And now it loads the characters correctly :-) – Doh09 Dec 25 '20 at 13:42
  • hi @Doh09 , thanks for the "tick" ! I suggest, just ask a new question - that's an interesting question. (On this site, the mods prefer that you ask separate new questions you know?) Merry Xmas 2020 ! – Fattie Dec 25 '20 at 14:32
  • Oh it's resolved already, so no need for a new question. Thanks though. Merry Christmas to you too. – Doh09 Dec 25 '20 at 14:36
  • 1
    Cheers! BTW I'm not sure if you looked in to it yet, but instead fo a Dictionary you could use a HashSet here (for even more performance, and simplicity). In general in C#, there are a large number of "collections" you can use in different cases. they *really* make things easier. there are many tutorials, eg https://programmingwithmosh.com/net/csharp-collections/ BTW you mention uppercase/lowercase. You may already know, if you want to solve that problem, fortunately it is simple. Simply save them all AS lc, and when you check one, convert it TO lc to check, "everything is lc"! Cheers – Fattie Dec 25 '20 at 14:45