I am writing a lexer, or tokenizer, for a programming language. One of the main functions is to split the line of source code into "tokens". I am achieving this by splitting on spaces to create string arrays. Therefore, when I want to preserve strings, I must temporarily change the contents to a keyword while the line is split, and then put the strings back in afterwards. This worked until I developed the variable system for the language, and I had to be able to preserve multiple strings. Then all exception hell broke loose.
Exceptions:
NullReferenceException (line 12) subStringArg[ini] = quotes[1];
IndexOutOfRangeException (line 34) value = value.Replace("STRING", subStringArg[ini]);
Minimum reproducible example:
public static string[] LexLine(string line)
{
string[] subStringArg = null;
string[] quotes = null;
string[] tokens = null;
int ini = 0; // Random name
while (line.Contains('\"'))
{
if (line.Contains('\"'))
{
quotes = line.Split('\"');
subStringArg[ini] = quotes[1];
}
if (subStringArg != null)
{
line = line.Replace(quotes[1], "STRING");
line = line.Replace("\\", "");
line = line.Replace("\"", "");
}
ini++;
}
tokens = line.Split(' ');
if (tokens[0] == "Output" && tokens[1] != "STRING")
{
tokens[1] = IO.OutputVariable(tokens[1]);
}
ini = 0;
foreach (string i in tokens)
{
if (i == "STRING" && subStringArg != null)
{
string value = i;
value = value.Replace("STRING", subStringArg[ini]);
tokens[currentArrayEntry] = value;
}
currentArrayEntry++;
ini++;
}
return tokens;
}
Source code (from my lang):
Output "Defining variables..." to Console. // Exception thrown here
New string of name "sampleStringVar" with value "sample".
Output "enter your name:" to Console.
Get input.
Output sampleStringVar to Console.
I'm asking here because I am clueless as what to do. I should not be getting a NullReferenceException from assigning values.
subStringArg?quotes[1]tosubStringArg[ini]. also I must clarify that when I debugged, the value ofquotes[1]had a value.ini = 0before line 34. When the exception is thrown,quotes[1]is not null, the only null variables aresubStringArgandtokens, andini=0.