Congrats on writing this great library! I'm thinking about integrating this into our (soon to be released) own open-source toolkit for privacy & security engineering, where we have a component for detecting personal and sensitive information for which the NER functionality would be super interesting. I'd love to use a Golang-only library like prose for this as our tool is also written in Go and integrating a toolkit like Spacy would drastically complicate the deployment of our tool.
How would one go about building a POS/NER model for other languages? I'm happy to have a stab at this and open a PR to add e.g. a German language model, right now I'm not entirely sure how to get started on this though. My basic understanding is that I could use e.g. Prodigy or a similar tool to label German text and then train a custom POS & NER model. Reading your code it seems I would also need to add a custom tokenization/stemming routing for German (and possibly a way to make this generric). Is that correct? If you can give me a few pointers and how to get started on this I'd be happy to help!
Congrats on writing this great library! I'm thinking about integrating this into our (soon to be released) own open-source toolkit for privacy & security engineering, where we have a component for detecting personal and sensitive information for which the NER functionality would be super interesting. I'd love to use a Golang-only library like prose for this as our tool is also written in Go and integrating a toolkit like Spacy would drastically complicate the deployment of our tool.
How would one go about building a POS/NER model for other languages? I'm happy to have a stab at this and open a PR to add e.g. a German language model, right now I'm not entirely sure how to get started on this though. My basic understanding is that I could use e.g. Prodigy or a similar tool to label German text and then train a custom POS & NER model. Reading your code it seems I would also need to add a custom tokenization/stemming routing for German (and possibly a way to make this generric). Is that correct? If you can give me a few pointers and how to get started on this I'd be happy to help!