FreeBSD FridayProgramming / DevelopmentWPF Databinding TutorialMy Posts

Welcome to my technology learning website. The goal of this site is to learn and retain as much information as possible. I consider this blog my knowledge base and hope that you both find and share knowledge here as well.Also for those wondering, Rhyous is pronounced ‘rī-əs. As in the words ‘Rye‘ and ‘us‘ compounded into a two syllable word.

Please take a moment to subscribe, or setup an RSS feed so you never miss a post.

Splitting sentences in C# using Stanford.NLP

So I need to break some sentences up. I have a pretty cool regex that does this, however, I want to try out Stanford.NLP for this. Let’s check it out.

  1. Create a Visual Studio C# project.
    I chose a New Console Project and named it SentenceSplitter.
  2. Right-click on the project and choose “Manage NuGet Packages.
  3. Add the Stanford.NLP.CoreNLP nuget package.
  4. Add the following code to Program.cs (This is a variation of the code provide here: http://sergey-tihon.github.io/Stanford.NLP.NET/StanfordCoreNLP.html
    using edu.stanford.nlp.ling;
    using edu.stanford.nlp.pipeline;
    using java.util;
    using System;
    using System.IO;
    using Console = System.Console;
    
    namespace SentenceSplitter
    {
        class Program
        {
            static void Main(string[] args)
            {
                // Path to the folder with models extracted from `stanford-corenlp-3.4-models.jar`
                var jarRoot = @"stanford-corenlp-3.4-models\";
    
                const string text = "I went or a run. Then I went to work. I had a good lunch meeting with a friend name John Jr. The commute home was pretty good.";
    
                // Annotation pipeline configuration
                var props = new Properties();
                props.setProperty("annotators", "tokenize, ssplit, pos, lemma, ner, parse, dcoref");
                props.setProperty("sutime.binders", "0");
    
                // We should change current directory, so StanfordCoreNLP could find all the model files automatically 
                var curDir = Environment.CurrentDirectory;
                Directory.SetCurrentDirectory(jarRoot);
                var pipeline = new StanfordCoreNLP(props);
                Directory.SetCurrentDirectory(curDir);
    
                // Annotation
                var annotation = new Annotation(text);
                pipeline.annotate(annotation);
    
                // these are all the sentences in this document
                // a CoreMap is essentially a Map that uses class objects as keys and has values with custom types
                var sentences = annotation.get(typeof(CoreAnnotations.SentencesAnnotation));
                if (sentences == null)
                {
                    return;
                }
                foreach (Annotation sentence in sentences as ArrayList)
                {
                    Console.WriteLine(sentence);
                }
            }
        }
    }
    

    Warning! If you try to run here, you will get the following exception: Unrecoverable error while loading a tagger model

    java.lang.RuntimeException was unhandled
      HResult=-2146233088
      Message=edu.stanford.nlp.io.RuntimeIOException: Unrecoverable error while loading a tagger model
      Source=stanford-corenlp-3.4
      StackTrace:
           at edu.stanford.nlp.pipeline.StanfordCoreNLP.4.create()
           at edu.stanford.nlp.pipeline.AnnotatorPool.get(String name)
           at edu.stanford.nlp.pipeline.StanfordCoreNLP.construct(Properties A_1, Boolean A_2)
           at edu.stanford.nlp.pipeline.StanfordCoreNLP..ctor(Properties props, Boolean enforceRequirements)
           at edu.stanford.nlp.pipeline.StanfordCoreNLP..ctor(Properties props)
           at SentenceSplitter.Program.Main(String[] args) in c:\Users\jbarneck\Documents\Projects\NLP\SentenceSplitter\SentenceSplitter\Program.cs:line 20
           at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
           at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
           at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
           at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
           at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
           at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
           at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
           at System.Threading.ThreadHelper.ThreadStart()
      InnerException: edu.stanford.nlp.io.RuntimeIOException
           HResult=-2146233088
           Message=Unrecoverable error while loading a tagger model
           Source=stanford-corenlp-3.4
           StackTrace:
                at edu.stanford.nlp.tagger.maxent.MaxentTagger.readModelAndInit(Properties config, String modelFileOrUrl, Boolean printLoading)
                at edu.stanford.nlp.tagger.maxent.MaxentTagger..ctor(String modelFile, Properties config, Boolean printLoading)
                at edu.stanford.nlp.tagger.maxent.MaxentTagger..ctor(String modelFile)
                at edu.stanford.nlp.pipeline.POSTaggerAnnotator.loadModel(String A_0, Boolean A_1)
                at edu.stanford.nlp.pipeline.POSTaggerAnnotator..ctor(String annotatorName, Properties props)
                at edu.stanford.nlp.pipeline.StanfordCoreNLP.4.create()
           InnerException: java.io.IOException
                HResult=-2146233088
                Message=Unable to resolve "edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger" as either class path, filename or URL
                Source=stanford-corenlp-3.4
                StackTrace:
                     at edu.stanford.nlp.io.IOUtils.getInputStreamFromURLOrClasspathOrFileSystem(String textFileOrUrl)
                     at edu.stanford.nlp.tagger.maxent.MaxentTagger.readModelAndInit(Properties config, String modelFileOrUrl, Boolean printLoading)
                InnerException: 
    
    
  5. Download the stanford-corenlp-full-3.4.x.zip file from here: http://nlp.stanford.edu/software/corenlp.shtml#Download
  6. Extract the stanford-corenlp-full-2014-6-16.x.zip.
    Note: Over time, as new versions come out, make sure the version you download matches the version of your NuGet package.
  7. Extract the stanford-corenlp-3.4-models.jar file to stanford-corenlp-3.4-models.
    I used 7zip to extract the jar file.
  8. Copy the stanford-corenlp-3.4-models folder to your Visual Studio project files.
    Note: This is one way to include the jar file in your project. Other ways might be a copy action or another good way would be to use an app.config appSetting. I chose this way because it makes all my files part of the project for this demo. I would probably use the app.config method in production.
  9. In Visual Studio, use ctrl + left click to  highlight the stanford-corenlp-3.4-models folder and all subfolders.
  10. Open Properties (Press F4), and change the namespace provider setting to false.
  11. In Visual Studio, use ctrl + left click to  highlight the files under the stanford-corenlp-3.4-models folder and all files in all subfolders.
  12. Open Properties (Press F4), and change the Build Action to Content and the Copy to Output Directory setting to Copy if newer.
  13. Run the code.

 

Note: At first I tried to just load the model file. That doesn’t work. I got an exception. I had to set the @jarpath as shown above. I needed to copy all the contents of the jar file.

Results

Notice that I through it curve ball by ending a sentence with Jr. It still figured it out.

I went or a run. Then I went to work. I had a good lunch meeting with a friend name John Jr. The commute home was pretty good.

However, I just tried this paragraph and it did NOT detect the break after the first sentence.

Exit Room A. Turn right. Go down the hall to the first door. Enter Room B.

I am pretty sure this second failure is due to the similarity in string with a legitimate first name, middle initial, last name.

Jared A. Barneck
Room A. Turn

Now the question is, how do I train it to not make such mistakes?

One Comment

  1. Vincent Mitchell says:

    Answer, rigid input structure requires 2 spaces after sentence, on after everything else. So now if your input is structured properly, the parser can distinguish it. Other alternatives for end of sentence apply, such as line break, etc.

Leave a Reply

Powered by sweet Captcha