NLP: What are some popular packages for phrase tokenization?

Question

I'm trying to tokenize some sentences into phrases. For instance, given

I think you're cute and I want to know more about you

The tokens can be something like

I think you're cute

and

I want to know more about you

Similarly, given input

Today was great, but the weather could have been better.

Tokens:

Today was great

and

the weather could have been better

Can NLTK or similar packages achieve this?

Any advice appreciated.

score 0 · Accepted Answer · answered Jan 20 '19 at 15:26

Spacy can do this. Spacy's semantic parser is based on Language models trained on large corpus of text.

This parser can break sentence into lower level components such as words / phrases.

More details and examples :

1 Answers1